We seem to have solved the memory leak problem by replacing a faulty module in the compiled version of... I forget? Apache or Linux. Then we solved another memory leak by downgrading PHP to 5.2x since some modules we are using have problems with 5.3 and above. Thanks Rackspace! for providing continued support for 5.2 when Zend and even Centos had abandoned it. Now we are trying to find the residue of the problem... and we are looking at possible IPSF - intermittent power supply fault.
Sufficient to explain symptoms but not necessary... it could instead be three other things, but this is the simplest explanation. Which means replacing the server since paying for datacenter techs to work on hardware is not cost effective.
IPSF was what killed ZsaZsa, our other server two months ago. Magda (the BC server) is a near-twin. ZZ's disease progressed from IPSF to RPSF (recurrent... which made it easier to detect and pretty much impossible to run the machine. ZsaZsa didn't have a leg to stand on, as it were. Actually, ZZ had a redundant power supply and a slight rewiring has restored her to usefulness at Piper's home, she's now our test bed. That name metaphor is getting a bit strained. :)
Anyway, it's going to cost $800 to $2000 to replace Magda and we are now debating on how long we should try to limp along.
Hugs,
Erin, Piper, and the gang
Comments
What about Eva?
...I understand she gets allergic smelling hay, but she does adore a penthouse view.
Love, Andrea Lena
Eva runs Stardust...
...Tuckerspawn, TGFiciton.net. et al. where her hayfever hardly bothers her at all. :)
Mama Jolie is the mail server and cousins Gigi, Pandoria and Bridget (ZZ's replacement) make up the rest of our proud little family group.
Hugs,
Erin
= Give everyone the benefit of the doubt because certainty is a fragile thing that can be shattered by one overlooked fact.
Sounds like
we need to go for a pay to view membership or loads of irritating pop up ads.
Angharad
Erin, you make it sound
like a plumbing problem. Is there some sort of malicious program out to destroy Big Closet and other similar sites?
May Your Light Forever Shine
Unlikely
Hacktivists usually do something simple like a DDoS attack whilst criminals would want the site up to use as a watering hole to distribute malware.
Persephone
Non sum qualis eram
Obviously need a more reliable power supply
I suspect OEM power supplies on some brands of computers are not that reliable. I know how to choose a reliable brand for a PC but for the life of me I do not know how to choose one for a server. If Piper can go ahead and research on say 'www.jonnyguru.com' sometimes they do reviews on server power supplies and maybe a replacement can be found.
It would be much cheaper than replacing an entire server.
Finally, I hope the power rating of the power supply is sufficient and if possible I would go for one with a bit more reserve if possible.
Kim
Just a quick FYI, OEM is our
Just a quick FYI, OEM is our only option.
These servers are "rack mount" servers. Specifically 1U servers. Each server is only as tall as your CD/DVD drive on a desktop. Many of the parts that go into these servers were actually designed for laptops, and are smaller and power efficient.
Add to that, the fact that to replace the PSUs, we'd need to order them either from the company that originally built the servers (Penguin Computing) or from someone on eBay who might be selling us "new" used hardware, and it becomes more cost effective to upgrade.
The primary cost in having your own server is physical space; electricity and of course hardware replacement are a close second with bandwidth after that.
-Piper
I've seen some on that website for yes
... rack mount servers. For the model reviewed I've found it priced for 130 for a 500W model. I've seen cheaper just googling for "Delta 500W power supply". I don't know how many watts you need but based on the the fact it only has 2 quad core CPUs plus no graphics card I am guessing 500W should be sufficient if it is at all power efficient. and yes we are talking about 1U compatible units. That has always been my assumption.
But question is, is it cheaper to have some ape replace the entire unit at the datacenter?
If you are going to upgrade I am hoping that you don't upgrade so that you need another upgrade in 2 years.
Kim
When we upgrade, we only plan
When we upgrade, we only plan for 1-2 years out of our servers. Magda gets beat to hell. 450w-500w is fine for our uses. One of the big issues is labour. It costs us $70/hr to pay the data center to do work on a server. But if we send them a new server, it's free for them to install it, and put it into place, and we only pay for shipping and handling of the old unit.
Right now, we aren't at full capacity of the server, but we don't have MUCH room to expand, and drive the site to it's next incarnation, so we would likely upgrade it in a couple months anyways.
But the BIGGEST thing that's stopping us from just replacing the PSU, is that we don't know for sure that it's an PSU issue.
Everything that's happening, says that that's the simplest explanation. We've even talked with others in the field, and they confirm the possibility. But as anyone that's dealt with IT knows, an intermittent power fault, can cause other components to fail also. When the PSU went on ZsaZsa, it took out 2 HDD's and we were afraid it took out the SATA controller also.
We've already pulled both the SSD and one of the SATA HDD's out of operation on this server, and started work for "just in case there is catastrophic failure". Honest answer is, I don't feel safe diagnosing something like this, on the other side of the country, and so my recommendation is to replace the server.
I will promise, once it get's here, I'll tell you if a replacement PSU would fix it or not :P
-HuGgLeS-
-Piper
once there has been a known power surged
problem with one component, example: the power supply or CD drive shorting out or sparking in the case, I am so leery that I will replace the entire machine - case and all, rather than play "i don't know" and pour money into something that keeps throwing me fits. I play it safe, and yes it is expensive, but I have much less downtime doing it that way.
I have had a CD drive take out a hard drive from shorting out!
Chucked the entire unit into the trash after disassembling the hard drive, removing the platers and destroy the platter plates against a stone grinding wheel. Same for the ram sticks.
Data can be retrieved from those should you try to throw them away in a landfill or garbage bin. Same for iphones and smartphones you get rid of - remove those ram sticks inside and the tiny crystal storage card out! Gazelle.com doesn't pay for your phone without the data card or ram missing - they want your info to sell.
Sephrena
replacing a power supply
Can't you replace it? Maybe this is my lack of experience with servers talking, but power supplies on desktops are generally pretty easy to replace and I really can't imagine one on a server being all that different.
Have delightfully devious day,
Arms too short :)
The server is in Denver, I'm in CA and Piper in NJ. Hiring someone else to do it is expensive and risky. We have thought a lot about this. See, one thing is, we don't KNOW that the problem is the power supply. It's too intermittent to tell.
We're going to need to replace Magda someday soon anyway, BC is a big site and really needs a faster machine with more memory.
Hugs,
Erin
= Give everyone the benefit of the doubt because certainty is a fragile thing that can be shattered by one overlooked fact.
The advantage to a widely-separated fanbase
is that you just might find one fan, who is as competent as an orangutan, close enough to the hosting company, to try switching PSUs for you.
Unfortunately not me; I'd give it a whack... and possibly take pictures of the whacking.
Ellen, 22nd level Necromancer of Threads
The problem is that most of
The problem is that most of the hosting companies won't allow you to enter the data center. Instead, they do the installation and getting it online, and then allow you to work on it remotely. The closest you could get is someone going by and picking up the server to work offsite.
What's the motherboard model? Does anyone know? Also, the case brand and model should be listed on the original invoice, and I'd bet I can get a replacement (new) power supply.
The issue with power supplies is while most of the motherboards have gone to lithium-polymer capacitors, the power supplies still use rock bottom crap electrolytic capacitors, unless you go with a good brand like PC Power and Cooling - and pay for it. Antec isn't a good brand.
I'll get a life when it's proven and substantiated to be better than what I'm currently experiencing.
Oooooh riiight
*facepalm*
I remembered someone mentioning that before, but somehow that particular piece of information just seemed to slip from my mind. Aw, well...
Have delightfully devious day,
Bad news or good news???
I've been out pricing servers (for use in something we have going on and not because of Big Closet's probs).
At $2000 the price is cheap. The servers (4) that we are settling on for our company are going to cost upwards of $37K apiece.
At a guess, you intend to move the hard drives and a few other things from the present server to the 'new' one in order to save some bucks.
Good Luck
Anesidora
EDIT: By the way T H A N K S for all the hard work. The site is moving right along now and all the download problems I was having are gone.
Hip Hip Hooray for the BCTS team.
Anesidora
I've priced those kind of
I've priced those kind of servers in the past. I didn't get the contract, but I sure as heck made their previous supplier sweat and drop his prices by 40%. He saw them as suckers; I was willing to put in the effort even knowing he had the inside track - I detest IT people that gouge customers.
I'd _love_ to have 37k worth of hardware for my sites :)
I'll get a life when it's proven and substantiated to be better than what I'm currently experiencing.
Reconditioned Servers
We aren't actually looking to port anything other than data, from the current server to the new server. Used HDD's never do us well. We are actually, in fact looking to move from SATA to SAS drives.
We work to lower our costs by purchasing properly refurbished units from trusted sources, and only after doing a LOT of research into what's available where.
We currently have a selection of Dell Poweredge, Penguin Computing Altus, and HP Proliant servers.
When we moved everything to the current data center, we were intending to actually purchase a half cabinet, and install a blade system that could grow with us, but we couldn't find a data center in our price range, that was willing to install 220 twistlock without us purchasing a caged section of the data center. (BTW, when pricing your blade cabinets, make sure you take wiring infrastructure into account)
Trust me, I've researched the big hardware, but we only run 1 website on this server. There's no need for a $37,000 piece of hardware for it. Heck, the "game server"/"world server" that was purchased for an outside BC project, only cost $1000 and is spec'd higher than the grid servers that make up a major virtual world service I know of.
Unfortunately, we can't use reconditioned servers....
Our IT people have their hands tied a bit since the Feds can demand certain things regarding the equipment. While Renae is gone, my signature has to go onto the block where it asks for the person authorizing the P.O.
You can't believe how difficult it was for me to sign the damn thing. At least I feel a little better about it since you mentioned SAS drives and that's what they are putting in the servers. (8 drives each and three 768 w psu's. plus a bunch of other stuff. incl 768GB 1600MHz DD-3 mem. (at least I understand a little about the memory). This is a special project and independent of our stuff. Hell we only have two servers for the business so this mobile system is bigger than our business system. Hardened, too.
Anesidora
Edit: OOPs I meant 760 w psu's. it's the memory that's 768.
Paying for a new server
I'm glad to contribute, but not this month because I have to reach a particular savings goal and that won't happen for another two weeks. Definitely in April.
Just thinking out loud... It says over on the right that The Kitty's goal for March is $2,000. Is a new server's cost already part of that goal? Or maybe you would like to increase the goal? Oh, and has anyone explored Kickstarter as an option for funding costs like this? It's all the rage right now. (Crowdfunding in first-world countries is probably not just a trend, but a permanent, significant force, IMHO.) I've been hearing a lot of stories of wildly successful Kickstarter campaigns for things like a new sewing machine for a small business and other projects that I would not have predicted would raise a lot of interest, nor money.
Great work on narrowing down the server issues. And yes, BCTS has been performing extremely well for me ever since the upgrade/rebuild/downgrade/etc. I'm going to contribute money, but also thank you for the skill and labor of love and devotion and your personal time for this site. BCTS is a home away from home for me. What would we possibly do without you?!
Annemarie
a motherly worrier
This is so weird
I just discovered Kickstarter last Friday.
I've seen one or two gender-squishy projects there (but I've only scratched the surface) and they did not seem to be doing too well.
But of course, I'd expect their customers to be representative of the same groups that comprise society at large so this is not a giant surprise.
However, as more and more open minded people learn about Kickstarter (and eventually it's competitors) projects relating to alternative lifestyles should start doing much better.
The future suddenly looks a little less dim.
T
Don't know if this would work for you, but...
...have you looked into Amazon Web Services (AWS)? We use EC2 server instances and S3 storage at work for the server side of our iOS/Android mobile app and, AFAIK, all our other server needs. I believe Netflix Streaming runs on AWS. Also I helped my son set up a one-year free trial to run a personal Minecraft server so I know it can be quite cost-effective across a wide range of use cases, if it does what you need. They have a cost estimate calculator here.
~Erin M.
No cloud!
Using cloud services is great unless you value your data, in which case you are now voluntarily placing your precious into the hands of amoral mercenaries.
As one possibility, if Amazon decides that A, they don't want to be hosting porn, and B, BigCloset is porn - then the entire website might go poof. And whether they would do that now means absolutely nothing; you'd need to know if they will do that at any point in the future.
As another, they could wait two years, then disable download-from and quintuple their rates, under the business theory "Whaddaya gonna do about it?"
Oh, and bets that Amazon wouldn't be data-mining the S*** out of everything they host? How much are you willing to bet?
Could be useful as a backup website - i.e., sync everything TO the Amazon, and then if this site/hardware goes blooey, you've got a running backup elsewhere. But there are problems, like database sync, with that, and I'm not sure it would be worth the significant effort and hassle. Or the money.
Ellen, 22nd level Necromancer of Threads
I looked into AWS myself, for
I looked into AWS myself, for the comic hosting that I do, and basically choked. If you have a low bandwidth site, you're okay, but if you do a lot of bandwidth (or a lot of processing), they'll eat your lunch.
Don't worry about your privacy, worry about your pocketbook!
I'll get a life when it's proven and substantiated to be better than what I'm currently experiencing.
All true...
...but I worked at Amazon a few years back and have some friends still there, and while in my experience they treat their developers like shit compared to most of the industry, they are dead serious about being customer focused. Their business model is basically to screw everyone EXCEPT their customers.
Things can change, though, and corporations are legally required to behave sociopathically, so yes, there is always that risk. And I admit I tend to err on the side of being too trusting.
That's some business model
In slightly other words, "screw almost everyone".
I do business with Amazon from time to time, and have never had any reason to worry about them. Knowing this new thing about them does not, in general, bother me. This is just how life is in a crony Capitalist/crony Socialist society. I don't have to like it, but I do have to live with it. Till enough people open their eyes.
***
Still, there does seem to be a lot of that going around, and I've been thinking I know why. So I was relieved to see that you had noticed it too (emphasis added):
I was beginning to worry that I was the only one who could see it. And of course that would mean *I* have a problem, not them. If you think about it for a minute, you'll see why the law and the regulations work this way. Politicians are in the business of steali ... um, re-direcing ... more and more of the money that flows past them. And sometimes the voters notice. When we do, the politicians need to be able to point to someone else.
Many corporation CEOs and many union Bosses really are doing bad things, so the accusations are valid in many cases.
IOW they weren't 'framed', they were 'entrapped'. They actaully did do the bad thing, but they did it because they thought the politician would always cover for them. They thought the politician would go down with them, if they got caught. Can you say gullible? I'd love to be a fly on the wall when some chairman of some board realizes she has been f*ck*d like this.
Oh damn - it's like chocolate. I have to begin a new relationship by giving them the benefit of the doubt. I've learned to compensate by not having a lot of relationships. But the ones I do have are awesome.
T
Cloud expense
At the current time, cloud hosting would increase the costs for BC by a factor of about 20. We do a small amount of cloud storage for static stuff like images and backups but this is really low bandwidth stuff and is hardly worth it.
What we are actually looking into is owning our own cloud. For about twice the cost of what we are paying now we could host a quad server with quick internal links to separate each of the functions of BC onto its own server. The server would cost about three times as much to acquire as a replacement for the current server but it really does seem a good direction to go. So, we're looking at it.
Hugs,
Erin
= Give everyone the benefit of the doubt because certainty is a fragile thing that can be shattered by one overlooked fact.
Uhm.
"What we are actually looking into is owning our own cloud. For about twice the cost of what we are paying now we could host a quad server with quick internal links to separate each of the functions of BC onto its own server. The server would cost about three times as much to acquire as a replacement for the current server but it really does seem a good direction to go. So, we're looking at it."
That sounds like a virtual cloud. Not a true cloud. True clouds are by definition based on the model pioneered by Google of using dozens of underpowered machines to do the work of one megalithic server.
Virtual clouds on newer multiprocessor machines DO tend to help distribute load... but... they don't distribute nearly as well or efficiently as a real cloud. Their primary benefits are as a transitional model to a true cloud when you already have the big megalithic server, and added security due to the sandboxing effect.
But then, a real cloud would probably cost even more to host because you'd have to buy basically an entire room, or at least a rack or two, instead of just placing a single coupla U server into a rack.
I dunno. I suppose you've done the research and are aware of all the tradeoffs of the different hosting styles you might consider for BC's future. I'm just not sure that a virtual cloud would give ENOUGH benefit for the cost. Especially since it would inevitably be yet another temporary measure and you'd most probably eventually end up needing to go true cloud anyways. And you DON'T already have the megalithic server handy. ;)
Abigail Drew.
Cloud computing wasn't
Cloud computing wasn't pioneered by Google, or Amazon. They just brought it to it's current 'buzz word' status.
Beowulf clusters predate both companies, and those were used as some of the original 'high availability' clusters. (And to do Titanic)
Linux-HA is an open source high availability clustering project.
I'll get a life when it's proven and substantiated to be better than what I'm currently experiencing.
All solutions are temporary
It's the nature of technology.
Hugs,
Erin
= Give everyone the benefit of the doubt because certainty is a fragile thing that can be shattered by one overlooked fact.
Atually it's a true
Atually it's a true cloud.
IT's one 2U rackmount box, with 4 motherboards, each with 2 QuadCore CPU's, each with it's own ram, and each with it's own dedicated SAS drives (although admittedly they are on a single drive backplane). It's basically 4 separate servers, in 1 box. It makes it cheaper to host due to the reduced space. And the need for only 1 power drop.
-Piper
Oh!
Awesome. Now THAT makes sense!
Thanks, Piper!
Abigail Drew.
Srsly?
Wow. I would not have expected that at all. Makes me wonder how and for whom it ever would make sense.
So, um, never mind.
It's all based on what you
It's all based on what you need. The old adage - use the right tool for the job - wasn't that a story. ;)
Admittedly, this is only my faulty memory at work so your mileage may vary but...
originally, cloud computing was designed to meet the needs of high processing power - i.e. lots of ops. (a lot of mathematical calculations or data processing operations) Also, some university distributed computing projects, preceding cloud computing, used to depend on the goodwill of people and had access to spare time on ppl's processors free of charge. Now, all these cloud services charge.
So if BC was trying to calculate the next prime number - that'd be lots of ops and maybe right for cloud computing. Cloud Computing has moved into the niche position also of being "always present" for backups of mobile machines - but not necessarily cheapest. Many ppl pay extra to know that their phone, ipad, tablet, etc is being continually backed up without having to dock to a physical computer.
BC's hosting prime data, which is a different creature and doesn't take lots of ops. Nor do the BC servers take a stroll at lunchtime, nor travel otherwise. ;) So they don't need cloud backup.
BC is imho, a pretty lean website. So guaranteed 24x7 uptime, fast retrieval of data, and bandwidth(prolly not a problem) would be the priorities to optimize. Coming from a database perspective - if this proposed "virtual cloud" means that all data can kept in database cache/memory or the increased in number of drives assists in reducing data retrieval time off the physical medium, I think it might be the right next step.
Of course, I think there's a lotta sys admin/dba ppl on here with more chits than I, at the moment. And my climbing skills and coordination are worse than an orangutan. =] So please correct me if needed.
Hope this helps clarify when cloud computing might be beneficial....
PS - I really hope you don't go to pay membership. This is my go-to website and a life-line for me since I don't get out into the community anymore. Although I'm one of "them" - f2m ;)
We're already doing some of the virtual cloud stuff
BC's main server is named Magda, this is the one that runs the database and serves the pages. Search and a separate search index database are run on a second server named Eva. A third server has been used in the past to serve static data, like images and files, but that is not being done at the moment, it's really not that big of a saving in load.
The big saving on load is that we cache the site for non-logged-in users which is why everyone's read counts went down about a year ago. Only about 1 in 200 non-logged-in readers are actually counted. Real read counts are 2 to 5 times as high as actually show up but that's complicated by how the system counts reads when someone is just reading comments. On a typical afternoon, we have 50-90 members on and 300-700 casual visitors.
We use between 6 and 10% of our bandwidth at peak, BW is not our problem. The bottleneck is database access and having a virtual cloud with the database actually on a separate machine from the front end but with a fast internal connection would enable us to continue to expand for, oh, another two or three years. :)
We've got a plan also for a sort of private cloud with servers in three locations, two being mirrors of the master site. And there are other options on the horizon.
We have no plans for making BC a pay site. We did experiment with some mechanisms for offering premium content but even so, the premiums under that plan would have been available to anyone writing stories, blogs, making comments, pushing the kudos button or even logging in regularly. That didn't work out, it shot the number of database accesses through the roof.
= Give everyone the benefit of the doubt because certainty is a fragile thing that can be shattered by one overlooked fact.
It's clouds illusions I recall...
...I really don't know clouds at all.
Love, Andrea Lena
I like to look at clouds
sometimes, you can see the shapes of bunnies and unicorns in them ......
Database Access / Mirroring / etc...
Cool - that sounds like a nice set-up. Congrats on the cheap colo charges.
Pulling the searches to Eva is good - Depending on how frequent updates are, you could even save common searches in views and keep 'em in cache - just trigger an update/rebuild when a new story is added... but hey, I bet you're already doing that. :)
Mirroring is a good thing, particularly with fairly static data. Are you using raids/multiple disks? I know between the combination of OS and DB that you used to be able to assign different DB and/or tables to different disks to ensure more parallel writes (& accesses). But to have 50-90 users hitting "potentially" the new stories, posting comments and updating read/kudos counts, that's a lotta locks on the same data and potentially the same tables and a lotta potential for deadlock...
Assuming that the casual visitors are allowed dirty reads, no problem - mirror & caching - will solve that like you've done.
A couple of the big DB engines had really good optimization reporting allowing you to get detailed info about the types of locks, length of time held and potential deadlock situations.... as well as query times, etc...
I'm sure you've looked into it and know all this, particularly since you re-did and optimized the reads.
But on of the biggest contentions in database access is during inserts or updates... Which with new stories, the read count & to a lesser extent comments are what will be hit frequently and depending on the locking scheme may have a large effect on access. Even more than tracking user access, etc.... I'm assuming you allow dirty reads, cuz it wouldn't make sense otherwise.
Typically, we don't want to be locking the same column/table, over and over and over but it often happens... Are you using multiple tables to track reads and then aggregating them, maybe into a view, for the story count? Using temporary holding tables and periodic updates? etc.
I'm assuming that you've already optimized the number of DB engines and listening ports, if possible.
If you don't mind, I'm curious now as to the solution you implemented for the reads.
Thanks for explaining more,
Arik
But, but...
...Netflix uses Amazon's EC2 for all their streaming video. That's a reasonable amount of processing, to be sure, but an astronomical amount of bandwidth.
Also as I understand it, AWS exists because Amazon has to have all this extra capacity to handle the Christmas buying frenzy, and wanted to basically rent it out the rest of the year when they weren't using it themselves rather than having it sit idle and unproductive. So it's optimized to handle the kinds of computing, database access, and bandwidth needs that Amazon has, which seem to me to be awfully similar (if on a much larger scale) to what a site like BCTS would need. So it would seem like Amazon, with its economies of scale and its obsessive focus on minimizing costs to its customers, ought by all rights to be able to offer just what this site needs at a far lower cost than other options, including DIY.
I'm not arguing that anyone is wrong about having found it not to be a good solution for their particular needs--just expressing my sincere bafflement that it turns out to be in fact such a spectacularly bad solution, at least financially, just where I would have thought it would do best.
Shows how much I know.
Scaling and pricing
All of the cloud stuff is priced to make sense if you have a large enough operation to benefit. We don't, it's really that simple.
Some of the cloud stuff is actually offered free or very cheaply at the low end. We're too big to benefit from that but not big enough to use the price break that really large scaling offers. The one really effective middle tier cloud deal we found still didn't quite work for us.
Also, on the pricing, the bandwidth charges are usually the same either direction. We do LOTS of downloads, not much upload. They offer cheap real estate which we don't need and charge heavy for downloading. Not much advantage for us.
One thing, we've got what may be the best deal in the world on hosting. It costs us about 1/5 to 1/3 what most colos charge. We had a good deal before, at about 1/2 of usual costs but now we have a great deal. It's hard for cloud computing to compete with that. Also, cloud computing is priced higher than coloing because there is no hardware cost and lower personnel costs so big companies save lots there. We get our hardware for about 1/10-1/5 of what the large companies do because we are buying their last generation stuff reconditioned. And we work cheap. :)
Hugs,
Erin
= Give everyone the benefit of the doubt because certainty is a fragile thing that can be shattered by one overlooked fact.
Do these problems have anything to do with...
why in the middle of this afternoon, after reading a story, I tried to go to the home page and I got a message which said the server was using an unsupported compression program? I was running Camino on a dual processor MAC. I can still reach Big Closet using my Safari program. My Camino is up-to-date according to the software update function.
I can move over to Safari, but all my bookmarks are on Camino. Any ideas?
Suzij
Compression program
We've been using that compression program for over two years. Not sure what went wrong on your end, but it is not a recent change on ours. Now we do do a reset at 1130 and 530 that will hang your browser if you are downloading at that time.
Far as I know, all modern browsers support the compression program we use. Including Camino.
Hugs,
Erin
= Give everyone the benefit of the doubt because certainty is a fragile thing that can be shattered by one overlooked fact.