Adobe research creates AI tool for transferring image styles between photographs

Posted:
in Mac Software
Users of Adobe's image editing software may see a new creative option added to the tools in the future, thanks to artificial intelligence research conducted by Adobe and Cornell University that can make changes to a photograph by transferring the style and other elements from another source.




The paper for Deep Photo Style Transfer describes the use of deep learning methods to analyze elements of a reference photo, acquiring information on the time of day, colors, weather conditions, and other items, reports The Next Web. This style can then be applied to a second image, changing elements to make it similar to the first, such as editing a dusk cityscape to one that appears to be an image captured in the middle of the day.

The researchers based their work on earlier algorithms that used a painterly transfer via a neural network, in a similar way. A Neural Style algorithm was then used on a target image to apply the style, but introduced various distortions into the image, making it unsuitable for photorealistic style transfers.

The researchers fixed this by constraining the transformation to be locally affine in colorspace, and to apply it as a custom layer that can be further adjusted. This approach successfully suppresses distortion and yields satisfying photorealistic style transfers in a broad variety of scenarios, according to the paper.

The recently-published paper does indicate Adobe is working hard to introduce artificial intelligence into its tools in the future, though it may not appear in Photoshop for a while, until Adobe can perfect the software's capabilities. In the meantime, code from the research has been made available to download in Github, so interested users can try out the tool for themselves.

Adobe's existing tools already use machine learning and AI in a limited capacity, announcing in November last year that it was introducing tools to automate tasks and provide extra assistance, powered by the Adobe Cloud Platform. For example, Adobe Sensei will help users in Stock Visual Search and Match Font, while the Liquify tool in Photoshop will be made face-aware.

The company has also teased the use of a voice-based assistant for basic image editing tasks. A proof of concept video showed a user cropping and flipping an image before posting it to Facebook, all by speaking to an iPad app.

Adobe believes it may be some time before such an assistant will be available for use by its customers.

Comments

  • Reply 1 of 12
    Pretty cool. Too bad they didn't use better examples though. I mean it's impressive that you can make a boring house look like it's been transported to an alien world with white grass and a brown sky, but how often is that called for. I'd rather see an example like described in the article "a dusk cityscape to one that appears to be an image captured in the middle of the day."
    SpamSandwichcornchip
  • Reply 2 of 12
    tmaytmay Posts: 1,716member
    Pretty cool. Too bad they didn't use better examples though. I mean it's impressive that you can make a boring house look like it's been transported to an alien world with white grass and a brown sky, but how often is that called for. I'd rather see an example like described in the article "a dusk cityscape to one that appears to be an image captured in the middle of the day."
    Infrared photography is an acquired taste, so a simulation of that is also. That said, being able to create a "sky" for a photo taken on a clear day would be a useful tool. Still, you would need to be able to simulate cloud shadows to best mimic a real photo, which this doesn't tool doesn't do. You would likely need a depth map in meta data of both the sky and landscape to be able to do that.




  • Reply 3 of 12
    spice-boyspice-boy Posts: 314member
    Am I the only one that thinks the "after" pictures are horrific?
    randominternetpersoncornchip
  • Reply 4 of 12
    polymniapolymnia Posts: 564member
    spice-boy said:
    Am I the only one that thinks the "after" pictures are horrific?
    I think you are. 

    If you feel the Reference photos are horrific, you will likely hate the Result images. 

    Its pretty safe to say that making a conventional image match a highly stylized image will work better than the other way around. 
  • Reply 5 of 12
    MarvinMarvin Posts: 13,902moderator
    spice-boy said:
    Am I the only one that thinks the "after" pictures are horrific?
    They look close to the reference. This would be an extreme example of how much they can change the source to become like a given reference. In real-world cases, people would just want to be matching composited elements into a scene e.g you crop a person out and paste them in somewhere and need the lighting/shadowing and color temperature to match the surroundings or you have a series of images that you need to match as a group.

    You'd be able to have the computer match portions of images to other images, something that would take hours to do manually. All these tools are just there to save time. They should apply AI to cropping too. Humans can easily see what an object is relative to a background but computers can't so humans have to slowly crop round the image. With object/shape recognition, the computer could do the job much more effectively.
    tmay
  • Reply 6 of 12
    MacProMacPro Posts: 16,187member
    So I want to use a reference pic of George Clooney for all my own photos ...  ;)  /kidding


    lolliver
  • Reply 7 of 12
    davendaven Posts: 406member
    Interesting. With minor tweaking you can mimic the work of other photographers. Automation is reducing the role of skilled people which is good and bad.
  • Reply 8 of 12
    SpamSandwichSpamSandwich Posts: 26,168member
    Could be very useful to filmmakers who need to match footage for scenes. Theoretically, these treatments would eventually be applicable to video in place of the laborious and costly color grading process.
    cornchip
  • Reply 9 of 12
    cornchipcornchip Posts: 731member
    Could be very useful to filmmakers who need to match footage for scenes. Theoretically, these treatments would eventually be applicable to video in place of the laborious and costly color grading process.
    The time saving would be unheard-of 
    SpamSandwich
  • Reply 10 of 12
    foggyhillfoggyhill Posts: 3,206member
    cornchip said:
    Could be very useful to filmmakers who need to match footage for scenes. Theoretically, these treatments would eventually be applicable to video in place of the laborious and costly color grading process.
    The time saving would be unheard-of 
    Yes, matching footage in a scene that's supposed to be continious, but was filmed say within an hour is a hellish, even footage filmed at the same moment from different angles may not blend well together! This would put a whole lot of expensive people out of business (everything related to color grading) and make indie filmmaking look even more pro like. Could help prolong the time you can film at dusk, or even make it possible to shoot golden hour shots at any time of day (probably need at least flat lighting to do that though :-).
  • Reply 11 of 12
    polymniapolymnia Posts: 564member
    Could be very useful to filmmakers who need to match footage for scenes. Theoretically, these treatments would eventually be applicable to video in place of the laborious and costly color grading process.
     It even theoretically. If this feature makes it into Photoshop, there are video editing tools built into it. This will be available for video at the same time it's available for stills 
  • Reply 12 of 12
    MacProMacPro Posts: 16,187member
    polymnia said:
    Could be very useful to filmmakers who need to match footage for scenes. Theoretically, these treatments would eventually be applicable to video in place of the laborious and costly color grading process.
     It even theoretically. If this feature makes it into Photoshop, there are video editing tools built into it. This will be available for video at the same time it's available for stills 
    One would think Apple could easily take the 'Match Frame' feature in FCPro X to the next level by allowing match to an image not just another video frame to the target clip with the same sorts of abilities.  If Apple don't then Premiere will for sure.  I've done this myself manually by pushing and /or reversing curves to make wacky clips then matching but it's very hit and miss and takes a long time.
    edited March 31
Sign In or Register to comment.