Generative AI and scientific images/diagrams

Generative AI for image generation is a controversial topic for many reasons.  Still, as someone who doesn't have a staff of graphic artists on hand to help make scientific illustrations, it has certainly been tempting to see whether it might be a useful tool.  My brief experiments are based using bing's integrated engine (which I believe is DALL-E 3) since Rice has a license.  The short summary:  Trying to make scientific illustrations this way is great at showing how this technology is not close to useful yet for this application.  While natively integrated w/ chatGPT, it still cannot parse prompts well, indicating that it doesn't really "know" the meaning of a lot of basic terms and how those relate to its image training data.

Let me give you an example.  When I asked it "Make me an image of a benzene molecule showing the chemical bonds between the carbon atoms", this is the response:  

When I tried to refine this by typing "Make an image of a benzene molecule.", it responded by showing me image search results with sources, which were all correctly benzene, but not an original image.  When I responded "Please create a new image (not from an existing source) of a benzene molecule.", it responded with "Ok, I will create a new image from scratch showing a benzene molecule." and showed me this:

I then tried more specificity in the prompt:  "Create (from scratch) an image of a benzene molecule, which consists of six carbon atoms (black spheres) linked in a ring, with one hydrogen atom (a smaller red sphere) attached to each carbon atom."  The result was this:

Closer, but I'm still surprised at how poorly any of these resemble the stock images of benzene that it correctly found through image searching on the internet.  That last one is particularly close, so I tried "The fourth image is close, but it should have only one hydrogen attached to each carbon, not three."  In classic chatGPT form, the response was "I’m sorry, I made a mistake. I’ll try to create a new image of a benzene molecule with only one hydrogen attached to each carbon. Please wait a moment."  It then produced this:

Clearly it just cannot understand or follow the instructions yet. 

Similar things happen if you ask it to make an image of a lattice.  If I ask "Make me an image from scratch of a simple cubic lattice.", the verbal and image responses are quite encouraging.  "I’ll try to create that. Here is a new image of a simple cubic lattice that I generated from scratch. A simple cubic lattice is a type of crystal structure where the atoms are arranged in a regular pattern of cubes. Each atom is located at the center of a cube and has six nearest neighbors. A simple cubic lattice is one of the simplest and most symmetric types of lattices."

This is quite close, though all of these have problems (non-orthogonal angles, uneven spacings).  Things go off the rails badly, though, if you want a face-centered cubic lattice.  It can say the definition of an FCC lattice reasonably well:  "I’ll try to create that. Here is a new image of a face-centered cubic lattice that I generated from scratch. A face-centered cubic lattice is a type of crystal structure where the atoms are arranged in a regular pattern of cubes. Each atom is located at the center of a cube and at the center of each face of the cube. A face-centered cubic lattice has 12 nearest neighbors for each atom and is one of the most common and dense types of lattices."  However, the images are, ummm, artistic.

These are quite different from the simple image search of the internet for "face-centered cubic lattice", which does a good job if you don't mind using someone else's image.  The bottom line:  These kinds of image generation tools have enormous potential to save us time, but right now the parsing technology and training methods just aren't there yet.  If a simple image search turns up the right targets, but the generation tool can't make images that look like those search results, then training has failed somewhere. 

If anyone out there has found some flavor of these image generation capabilities that really does work well for helping to make scientific illustrations, please let me know.

Update:  Google just released their free image generation tool, and while it still has problems like those above, it's pretty impressive.  For example, here is what it made from the prompt "drawing of a benzene molecule, consisting of six black carbon atoms arranged in a ring, with each carbon atom connected to one red hydrogen atom":

That's certainly better than anything DALL-E made.  Of course, the other three images that came along with that one were all screwed up.  Still, progress.