Mint tea is a favourite among herbal tea drinkers and is one of the easiest to grow yourself. The plant is very hardy and may even get out of control in the garden unless you take care to contain it. Mint should get a least a partial day of good sun, but all-day sun might be too much for it. Make sure to water well during the peak of summer heat. There are many varieties to choose from, each with it's own unique taste: spearmint, peppermint, apple mint or even chocolate mint. Mint will grow readily indoors.
Your tea is made with the small white and yellow flowers of the chamomile plant, rather than the leaves. There are two kinds of chamomile (German and Roman) and it's the German variety that makes the best tea. Chamomile likes sandy soils and lots of sun, but you'll need to give it plenty of water during the hottest parts of the summer. Though technically an annual, chamomile goes to seed so readily that you will likely see it every year in your garden. You can grow chamomile in containers on a balcony, but it doesn't do well indoors.
The lemon balm plant is actually closely related to mint, but has a distinct lemon aroma. It likes somewhat dry soil and partial shade during the day. Besides making a nice herbal tea, you can use lemon balm as a spice when cooking (I like it with fish). Like it's minty cousin, you can grow lemon balm indoors.
Rose hips will make a nicely citrus-tasting tea that is rich in vitamin C. Any rose plant will create 'hips' but Rugosa roses produce the largest ones. The hips are actually seed pods that form at the base of the rose blooms. When making tea with dried rose hips, you should slice them in half before steeping. You may want to remove the seeds before making your tea, but it's not necessary. If you do choose to de-seed your hips, do so before you dry them. Rugosa roses are hardy and cold-tolerant. They grow in bushes between 2' to 6' tall. These roses will grow just about anywhere, but aren't really suitable for a windowsill garden.
Lavender makes a lovely addition to any garden, even if you're not using it for tea. Lavender will grow 2 or 3 feet tall which makes it inappropriate for a windowsill, but can easily be grown on the balcony in containers. Your soil should be well-drained and lavender likes plenty of direct sun. Some lavender varities take the cold better than others. You might not think of lavender for tea, but it makes for a floral tasting tea that also blends well with other herbs (like chamomile).
There are several kinds of fennel, but the type typically grown for tea use is the sweet fennel. When dried, the seeds have a very strong licorice flavour. Unlike the other herbs, you don't really harvest fennel periodically through the summer. The plant will go to seed at the end of summer or start of fall. You can let the seed dry right on the plant and then collect for tea. You won't want to grow fennel indoors because it can grow up to 6 feet tall. Also, don't plant it next to it's close relative, dill, because they can cross-polinate. Fennel likes lots of sun and lots of water.