Make a cover: Whether it's a first day cover with your own personally produced artistic cachet, or a commemorative cover with a topical special cancel from a flower show and rose stamps you have chosen especially for the occasion, you have made your own personal piece of philatelic art for little expense. Of course, if you don’t want to make your own cover right away, why not go to a first day cover ceremony to see what others are doing?
Get in the holiday spirit: Google to find where you might get envelopes for holidays and occasions special canceled and forwarded to friends and loved ones. "The North Pole" for Christmas and "Love, CO" for Valentine’s Day are but two examples of cancels obtainable for the price of a stamp. Even non-philatelists will get a kick out of receiving such covers from their "stamp collecting friend": you.
Write a letter to a philatelic newspaper/journal: The amount of information and subjects in the philatelic world can be humbling and there are many questions that grow out of unusual stamps in your collection. I once questioned a Philippines stamp with the overprint: "'Help me stop smuggling' - President Marcos." Did the president have a smuggling problem he was trying to kick? If not, pretty odd wording there. Many experts responded to my letter, which had been published in a large stamp newspaper. The many answers I got to my question was like a human Google chain.
Meet those who make stamp art (mail artists): I was lucky enough to sell a mail art article to a major philatelic magazine. That was great, but the real fun was meeting -- through the mails of course -- the interesting people that are a part of this branch of stamp collecting. It is easy to start a collection of these items: those that produce them will be glad to help you get started.
Become a part of the stamp biz -- for stamps: There are still dealers around who will hire interns willing to work for stamps. You may wind up doing paperwork and other non-philatelic chores, but you will be working in a philatelic environment. And you'll be paid in stamps!
Get your family involved: Beyond the beginner practice of having them save envelopes for you, your family might find enjoyment in being included in your philatelic pursuits. When the cover flown by the space shuttle was sold by lottery by the USPS and individuals could only send in one order apiece, I got all family members to place orders to increase my chances of obtaining one of the sought-after covers. (Benefits: I was lucky enough to wind up with several covers. My family learned something about the collecting world and understood my odd pursuit better.)
Learn more about your favorite corner of the world: Stamps are ambassadors of culture and information. They can also satisfy the urge to visit a country when the expense of doing so is prohibitive. Some advanced collectors even plan a vacation around stamp shows and visits to dealers. I never went that far, but I did get a friend to get town cancels on stamped postcards from many post offices during her travels around Ireland. Matching the cancels to a map was a great way to recreate the trip's memories for her, while it got me some nice Irish stamps and taught me about a place of interest.
Remember: Collecting with passion is the best revenge.