FOLIAGE STUDY

Do you struggle like a lot of artists with creating foliage? Then this is the article for you! Sven joins us for this foliage-packed article where we cover his full workflow of creating stunning foliage assets. From sculpting in ZBrush, texturing in Substance Painter, constructing and assembling foliage meshes in Blender as well as lighting and presentation tips, Sven’s got you covered so you never have to fear creating foliage again!

Intro

Hey, my name is Sven Mrđen and I am working as a freelance prop / environment artist and outsourcing manager. I’m from Zagreb, Croatia and my entry into this industry mostly happened by chance. My studies weren’t focused on 3D and it was just something I enjoyed doing on the side for a while, until one thing led to another and I found myself in the position I currently am today. There are visual artists in my family, but I have more of a musical background, so my involvement with the world of visual art mainly started during my college days when I also got into 3D graphics.

In this article I’m going to show you how I made some of the plants from my ongoing foliage study project, and what I learned along the way. The tools I’ll be using for this are ZBrush (for sculpting high poly elements and texture atlas creation), Substance Painter (for texturing), Blender (for assembling the plant) and Unreal Engine (for rendering and final presentation). So, join me in this breakdown and hopefully you will find something for yourself in it.

Gathering references

At this stage, the goal is to understand how a particular plant grows by analysing it through multiple lenses. Pay attention to as many parameters as possible and apply them to all aspects of the plant, from micro to macro. For example, look at the growth pattern and take a note on how branches and veins curve and split from bottom to top. Analyse direction and rotation of each element across the plant, analyse changes in scale and in which direction they happen, look at colours and aging and areas where colour shifts occur. Study the density of the plant and distances between veins / leaves / stems. Gather closeup references to have a good view of damaged areas and to see what other kinds of small details can be added in the texture. Look at the subject in many different ways and apply the same lenses to each element - veins, leaves, branches, main stems, barks etc.

When searching for references on the internet, it can be useful to search by scientific plant names, or even names in a local language. Sometimes I even use YouTube, as some videos can provide nice views from multiple angles. If it’s hard to see the plant’s growth pattern or what it looks like near the bottom when it’s often covered by leaves, you can look at botanical illustrations. They can also be good references for hand painted textures.

But oftentimes, the best references are the ones I capture myself, because I can focus the camera on smaller hidden things that can be challenging to find on the internet, and just by looking at it in real life helps to truly understand the nature of the plant, or any other object.

After that, I gather some 3D references to see if I can find how other artists have approached the plant. If I know I’ve seen a good example in a video game, I like to inspect it in the level, with photo mode if there’s an option for it. I think it’s one of the best learning resources, at least it has been for me. Finally, I recommend browsing through the Megascans library to see if you can find the plant there, or at least something similar to it. Being able to analyse scanned models and textures is invaluable and they can help a lot as a texturing benchmark, not just for plant creation, but in general.

Sculpting

When it comes to high poly, I’m sculpting everything in ZBrush. It doesn’t really matter how you create blockout meshes for leaves and petals. They could be simple sub-d models made in another 3D package, or you may just immediately start sculpting from a sphere in ZBrush and moving its parts around until you match the reference, but I like to quickly draw a black & white image with basic silhouettes by painting over photo references and use it as a mask from which I will generate starting geometry in ZBrush. It’s just a fool proof way of quickly making base shapes and ensuring proportions are perfect every time.

After setting up thickness and smoothing out the edges, I begin working on interior volume. One thing I sometimes notice on beginner leaf sculpts is that the veins are made by directly carving into the leaf with a wide soft brush while the areas in between remain intact. If we apply a normal map matcap, there is no gradient anywhere to be seen, and so the leaf ends up looking flat. My approach to it has been to first define the placement of veins by drawing lightly on the surface and then sculpting the volume around the veins. The choice of brushes is up to personal preference, a lot of them can do a similar thing, it’s just different flavours. A simple Standard or Inflate brush can be enough for most of the sculpt, and then using a brush like a Clay Buildup can help to introduce some irregularities to the surface. So, the way I look at it is that the veins I drew are there just to represent borders between individual volumes of the leaf - the veins themselves end up being covered and thinned out at the end.

I have been asked a lot about how to achieve the “meaty” look of the leaf. I think the thickness is a product of this definition of volumes and modification of the edges so they also have a noticeable gradient, because when you only have a flat image of a leaf, that would be one of the main things that would suggest how much depth it has. Here’s an example with especially thick leaves - notice the light value shift on the edges where the surface curves. Not every leaf will be this thick, but I like to make my leaves thicker than usual, for the same reason bevels are often exaggerated on hard surface models. One way to achieve this is by using the Move brush and pushing the geometry towards the edges.

I usually leave all the microdetail and damaged areas for the texturing stage. That way I have more control of the final result and it’s easier to iterate and change the amount of damage any time. At the end, I refine the overall shape of the leaf with a Move brush and either repeat the process from scratch for other leaves or if I want to finish it faster, I’ll duplicate the leaf and tweak it to get variations.

Most of these plants so far didn’t have wooden branches, but I’ll still mention how I did them on other projects - in ZBrush, after blocking out shapes with ZSpheres, I remeshed them to get a clean topology. Then, I exported and unwrapped them in Blender, using a tileable bark texture for preview. After that, in Substance Painter, I used the bark texture to create a displacement map for the entire branch which I eventually used to get a high poly branch that only required some additional clean-up in ZBrush.

When the sculpt is done, I export normal and opacity maps, as well as some additional ones like ambient occlusion and cavity, which can also be helpful during the texturing. One thing to consider at this point is how many individual pieces of each part of the plant will be needed to achieve a desired result. I think it depends on the complexity of the plant and how many elements need to fit in a single texture, how much visual variety does the plant have in general (which may either require more leaves to cover all the cases, or a couple of leaves may be enough because all of them look mostly the same), and also how many stages of leaf’s life cycle I want to showcase.

At the beginning I didn’t include a lot of variation in the texture, but in recent examples, where possible, I’m trying to create a balance of healthy green leaves (which will be used to build a majority of the plant), transitional leaves (which have some complementary colours), partially brown and completely brown and decayed leaves.

When I was working on Bergenia and gathering references, I noticed that some leaves were rotated, showing brighter desaturated colours underneath them. I thought it was a detail which would be a great addition to the overall colour palette of the plant so I included one of those in the texture as well. But then, I saw there were leaves that were cut in half, which was also interesting to me, and the total number raised by 2 more, 9 in total. On the other hand, I had only 3 versions of leaves (clusters in this case) on the texture for Dasylirion because the differences between individual leaves were negligible and due to their shape, there wasn’t enough resolution to showcase details on them, so it didn’t matter either way. So, organization of the texture sheet depends on a case-by-case basis.

Speaking of Dasylirion, I would like to stress the importance of planning ahead and considering the scale of individual elements when sculpting, because it may not be worth it to add details if they won’t be seen on a finished model. Dasylirion’s leaves are slender and they have thin straight fibers running through them, which end in a brush-like bract. A bunch of tiny golden spines are spread along their margins. Only after I sculpted the entire thing did I realize I’m going to have trouble fitting the leaves in a way so that those elements take up enough texture space - they were just so small. After going back to make some changes and enlarging them slightly, I managed to get good enough bakes, but at a cost of less cluster variations in total. This is why it’s a good thing to block out the texture atlas in advance, especially for more complex plants. It can be a simple drawing, focused on a silhouette. That way you can see how the details like these will read on an assembled model as soon as possible.

Texturing

With a well defined base sculpt and baked masks, texturing should be off to a good start. I texture the plants in Substance Painter. It enables me to combine both procedural and hand painted texturing workflows so I prefer it in this case.

The reason why I decided to create textures by painting them manually is that I’m not interested in pure photorealism in this case. I have always appreciated foliage that had a handcrafted look, even if they were leaning towards realism. That balance between realism and stylization of certain elements is something I want to keep working on, so even if during the texturing process I do try to get close to my photo references, the textures will never be perfectly photoreal just by the nature of the workflow. Of course, realism could always be pushed further with enough invested time, but that is not the point. Finally, I’ve found out I just enjoy the process of hand painting, which I haven’t done a lot of in the past. It was exactly this that sparked my interest in 3D foliage in the first place.

The way I approach the colouring is different each time as a lot of it depends on a plant, but basic ideas are the same, so I’m going to explain how I textured leaves for one of the earlier pieces, Hosta plant. This method won’t apply to every plant, but you should get some clues what to think about when texturing your own leaves.

I start by applying a darker green base colour. On top of it, I’ll be adding brighter details that will make things pop and darker lines on sculpted cavities to add visual depth to the leaf. Think of this as adding a bit of baked lighting information to the colouring. Since I haven’t added any micro detail during the sculpting stage, before continuing with the albedo, I’ll make sure the leaf reads well with a simple colour, so I focus on two of the things which I think are equally important as the albedo - normal and roughness information. Here’s an example of 2 leaves with a simple single-coloured albedo - one of them doesn’t have any normal and roughness variation, while the other one does.

As you can see, normal and roughness maps also affect the visual perception of colour, so even if the albedo is completely flat, the plant’s surface will still look detailed, while it will also react to lighting. In my experience, adding this information is a key step that will push the texture to the next level and in the beginning when I was figuring out these things, something always felt like it was missing until I added this kind of micro detail to the mix.

I achieved this in different ways during the study as I experimented with various workflows. Try to find a material with a similar pattern and only use its height map. The first time, I made it with a marble grunge texture, then it was with a leather material and once it was with a stencil projection of a premade leaf pattern mask based on a photo.

Back to the albedo, sometimes I may add a bit of color variation masked with some grunge maps, just to break up the surface right from the start. This layer can also affect the roughness channel. After this, things can go in many directions which, again, depend on the shape and positioning of the leaf. It could be darkening the bottom near the stem and along the middle to simulate some occlusion due to its curvature, or finding other ways to create some tonal shifts and gradations. I may also add brightness right at the border all around the leaf to simulate the lighting that is caught on edges of objects as they change angle.

Another way to create these details is to use some of the textures baked in ZBrush, like cavity and ambient occlusion. One trick you can do is to first use, for example, an AO map to bring out bright areas, then on another layer use that same AO map, but inverted, to add darker areas in the opposite places. Edit the levels and blur as necessary. And in situations where you may not be able to achieve good results painting by hand, I suggest getting familiar with stencils and texture projection from photo sources. Combine workflows if needed. Finally, since Painter can import Photoshop brushes, you can play with them and you may find a particular texture that will make painting certain details a breeze and introduce additional visual interest to each brushstroke.

Try to give each leaf its own hue so you can create different compositions of colours when assembling them into plants, but be careful, because it’s easy to overdo it - they still need to work well together. It is a good idea to start assembling the plant after blocking out the base texture and import it in the final render engine before detailing the colors, so you can preview the texture in the scene while working on it.

With the base texture done, at this point I’m working on smaller details, like spots of yellow, brown or white. This is one of the things I eventually automated with procedural grunge masks.

Finally, I work on damaged areas, which is the point where things start getting interesting. There are still times I’m not sure if the texture will work until I add this element to the leaf.

As I said before, I don’t sculpt damage in ZBrush and the reason is that I want to keep the control and flexibility of its placement. On a separate layer in Painter, I can directly remove opaque parts of the leaf. It’s just an opacity channel that sits on top of the layer stack and its opacity is set to 0, making everything completely transparent. The black mask is added to it, effectively turning it off and I can make it show up again where I want to by painting the mask with a white brush, thus creating holes and parts that are cut off. This is an example of painting the damage, coupled with some procedural elements that will also affect the area around it:

When I export texture maps from ZBrush, sometimes I leave an empty space on the side for a trim that I will use for stems. Then, in Painter, I draw an edge to edge rectangle in that area and do a quick colour pass, while the rest of the micro detail is already there since it is generated by previously mentioned procedural noise that gets applied everywhere. SSS mask is made by using previously baked maps and anchors from the layers below to mask out the veins, stems, branches and other smaller details. In the end, I always add some post processing effects to all of my textures, mainly sharpen and contrast filters.

Assembling

So far I assembled all the plants by hand, in Blender. I have been avoiding procedural tools partially because I didn’t want to end up being dependent on them, but also because I didn’t find them necessary at the beginning when I was just getting into this and was making simple ground cover plants. With more complex vegetation, assembling final plants did start to become more time consuming, but with my current workflow it still remains a relatively fast part of the process.

One of the key questions I’ve been asking myself when approaching bigger plants is how to deal with the optimization while maintaining a volume so it visually holds up even from close up camera shots and all angles. I’ve seen many approaches out there and have tried different techniques, but this time, with LOD0 I decided to go for something which I would “simply” call a “more optimized version of penultimate Megascans plant LOD”. That is, the one just before they become simple crossed cards. All the major parts of the plant are usually on their own separate card, but with an additional optimization pass and a combination with the usual baked clusters in some cases. I’ve seen a similar workflow being used in Middle-earth: Shadow of War and Ghost of Tsushima, and there’s a great presentation where this approach is explained in detail, which you can watch here:

That being said, I think it is something which depends on each plant, its importance in the scene, how much it will be scattered, its shape and the number and size of leaves, as each one requires at least a slightly different approach. What I found out is that a workflow where a large part of a plant is baked to a card, provides better results when the plant’s shapes don’t have so much volume and groups of leaves are mostly flat by nature, or it’s a cluster with a lot of thin and small individual pieces. But when it comes to plants where each leaf is relatively large and pointing in a different direction while it is rotated horizontally, it gets harder to make them look right and they usually look good only from certain angles, particularly the sides.

So this was the approach I took with Bergenia and Euphorbia and each leaf and flower were baked on their own card. In some other cases with thicker shrubs, I did a combination of individual geometry for leaves and flowers that are sticking out, while I used planes with larger baked clusters to fill up the middle where they won’t be seen so clearly.

Additional benefits of a more modular approach are better shadows and increased depth of plants, more resolution for each plant element, more flexibility and less time spent on refining baked clusters since it’s easy to make changes directly on the model at any time - but I don’t think it can be feasible in every situation, so I would still think twice before going so modular, or at least consider combining both workflows - otherwise triangle count could skyrocket.

After importing textures in Blender and setting up a simple shader, I apply them to a plane and cut out all the elements from the atlas, making sure to minimize transparent space on each card and only adding additional edges where needed so I can curve the cards to achieve a silhouette that is matching the reference.

In some cases, elements of the plants are flat enough that I only use cards with 1 to 2 triangles. What can also be done, is to reduce details on cards that will be partially hidden and favour more geometry on exposed areas.

In Blender, you can use the Decimate modifier to optimize the cards further. It can be useful as a regular clean-up pass after modelling leaves. In this example situation, after adding edges to improve the silhouette, a lot of the geometry that remained in the middle is unnecessary. By using Decimate, vertices in the middle are welded together, without affecting the silhouette of the leaf. This modifier can help to create LODs as well, together with manually removing smaller parts that won’t be seen from far away.

Since I’m not using SpeedTree here, modelling the plant can be somewhat time consuming, but if we consider the fact that the structure of a plant is made from repeating patterns, we can focus on a single building block and then reuse it as many times as necessary. I usually make a small group which consists of a big, medium and small shape. Then I duplicate that group over and over again into more complex groups while doing some manual tweaks on top to avoid obvious repetitions. Shapes on the edges are the smallest to achieve a smoother transition to the ground.

There are a few things to consider here - if we use linked duplicate method instead of normal duplication, that makes it possible to make edits to the mesh that will be applied to all linked meshes in the scene, which can be useful when we want to tweak the shape after already assembling the plant. In case you want to introduce more variation and remap some of the already assembled parts to a different part of the atlas texture, it’s also possible to use random selection and then repositioning only those UV islands.

The flowers on Bergenia were made by modelling a single flower with a stem, then duplicating it in a small group, putting that group on its own stem and then duplicating the entire thing across the main stem. To make it easier, the pivot point of each modular piece is located on the bottom of the stem. Then, by enabling the Snapping and Snap during transform options, we can ensure that flowers are always fixed to the target stem while being moved around. Finally, the entire finished stem can also be duplicated around and rotated for more variation.

Pay attention to the emerging silhouette of the plant when assembling it. In Blender, enabling Diffuse Colour render pass in Eevee will remove all the shading and textures from the models and show them in a clear black and white view. You can use this view mode to have a better look at the shape of the plant. Try to break up the shape to achieve more visual interest and make it look unique from different angles. This will also help later on, because just by rotating the plant around the scene, each instance of it will look slightly different.

When putting the elements together, sometimes small differences in positioning and angles can have a large impact on the overall look. In this case, if the leaves are angled too much towards the ground and are very close to each other, they won’t create as many shadowed areas, causing the entire cluster to lose volume and appear flatly lit.

Presentation

Most of the foliage I’ve showcased so far was rendered in UE5, using Lumen for the lighting. Before I explain the lighting process, I’ll just mention that as far as master material setups can go, I have kept mine very simple for now. It is just a standard foliage shader with some controls for the hue, saturation, roughness and normal intensity.

The main thing I’m trying to do when setting up a shot is to “sculpt” the volume of the plant with light and enhance its shape. Apart from the the usual directional light and sky light, I’m using supporting lights at various places, where each one has a specific purpose - separating the plant from the background in order to make the silhouette more readable, brightening the focal point, providing additional bounced light from the sky, accentuating complementary colours or adding some coloration to the surrounding environment. Before all of this, I make sure the plant works and looks good with a simple lighting setup, under a default sunlight. But for the sake of a better presentation, and since I’m mostly showcasing each plant on its own, I like to make things interesting with a few additional lights.

Here’s an example of my lighting workflow. This is one of the more complicated setups, the majority of the renders you’ve seen so far were lit just with 1-2 additional lights. As you may notice, I went for more of a studio setup by starting with an almost completely dark scene. There was just enough sunlight there to provide a low key base and avoid large pitch black areas. I’m usually using longer focal lengths like 70mm for these camera shots, with shallow depth of field.

Roughness of the surface can be controlled by the size of light sources. Lower light radius will sharpen the roughness variation, while higher values will make it flatter. Notice the difference this can make in the following comparison - low light radius value on the left and high light radius value on the right:

It’s important to keep in mind though that this will affect the sharpness of the shadows as well. You can combine smaller and larger light sources to control the composition and accentuate surface details.

Learning resources

Personally I learned a lot in general by reverse-engineering other models and projects and analysing assets in games. For tutorials and courses, I recommend checking out game-ready vegetation video tutorials on ArtStation and reading articles about foliage production from other artists.

Areas Looking To Grow As An Artist

I’ve been thinking about this question for a while and what I can say is that, I obviously want to keep growing and improving my technical skills as everyone else, from sculpting to lighting, but I’ve been feeling a strong need to create more meaningful personal art and try to reach people outside of this industry. Since I started doing this professionally, all of my personal projects also became “technical studies” in a way, which usually resulted in scope reduction in favor of more time investment in a specific part where I wanted to improve a particular skill. All of it was still worth it, but at the cost of meaning of the final piece. So, I’d like to take a step back and focus on a bigger picture again for my future projects, where I could also put into work more skills of mine, not just 3D production. And finally, achieving a unique and recognizable voice and overall art direction is something I strive for, although that is still probably far away and will come with time. Until then, I will keep experimenting.

Inspiration

Inspiration comes unexpectedly from various sources, I would say to just keep your eyes open and expose yourself to different things. Don’t limit yourself to the media you usually consume. I appreciate the fact 3D graphics opened my mind to new ways of looking at the world and noticing things I haven’t noticed before.

Feedback

I think having access to people who can give good and honest feedback is one of the most important things one needs to grow, especially as a beginner. I always suggest everyone asking me about this to get involved in communities such as Experience Points and The DiNusty Empire, as they host a ton of brilliant artists that are always willing to help. Without their support I probably wouldn’t be doing this today. As for me personally, I can’t say I really ask for feedback that much. I’m not very active on social media in general and don’t post often, so when I occasionally post some work in progress, I can usually tell just by the amount of engagement if I’m going in the right direction. It comes at a cost though, as the learning journey can feel lonely at times.

Additional advice

If there was only one thing I could say, it’s this - don’t let the perceived success of others demotivate you. Instead, find inspiration in their work, but focus on your own journey. As long as you see personal progress you’re probably doing fine. It is something everyone experiences to a certain degree and I think it never goes away, but it can get easier to deal with once you become more confident in your own skills and knowledge. And when you reach that point, stay humble.

Future Work

I have a few other ongoing projects I plan to continue working on after I’m finished with foliage. One of them in particular is vastly different from my past works and art style, and it is something I’m using as an experiment of sorts, so I’ll see where that will lead to.

Outro

In the end, I hope you have learned something from this article. Thanks to Experience Points for this opportunity and the great questions that reached beyond pure technical workflows. This project isn’t over yet and I’ll have more things to share, so if you want to keep following my work, you can find me on previously mentioned Discord communities, ArtStation, Twitter and LinkedIn.