Still don't feel "parent-y"

We just moved to a new house and we've found ourselves smack in the middle of a GREAT block with a lot of families and kids in almost every house, we feel very lucky to have found a place where we can really see our daughter growing up.  So for the first time I am hanging with a group of Moms, Dads, and kids that range in age from 1 to 13.  A few days ago our 4 year old neighbor came into our house for a playdate with my 2 year old, and I was struck with a sudden insecurity.  What do I know about a four year old?  What is appropriate?  Do I need to make up some rules? What do I do if she breaks them?  Auuggghh!!  I am not old enough for this - I can't be responsible for children!!  Babies - OK.  Children - Yikes!!  Isn't there another adult in the room?
How can it be that I am almost 40, yet I don't feel old enough to be the adult responsible for a group of kids?  I am the youngest in my family - so I grew up without really being around kids younger than me.  Growing up I babysat a ton, and was a nanny for a few years during college - so when I was younger I had no problem taking care of kids.  Maybe as I've gotten older and wiser I've learned the myriad of ways you can mess kids up...so each interaction takes on a larger perspective?  Maybe as I've figured out just how much I don't know I've gotten less confident in my actual knowledge?
Whatever the cause, I somehow don't feel the parent role beyond being the mother of my daughter (which comes very naturally to me).  I know I can take care of kids and I am responsible, vigilant, and capable...but I don't feel the part of the "adult".  You know...the old person in the room with the eyes in the back of their head ready to bust you for any infraction.  I am sure kids see my age and would give me that authority and respect if I needed to command it....but I just still feel so young.  Does everyone?