Results matching “time capsule” from Bill's Words

It has been a good day. That’s compared to yesterday, which was a not-so-good day. And the day before it, that one was a not-quite-bad day. The day before that? That was a horrible day. The day before that day? Oh, that one was superb.

And thus began the roller coaster ride that is cancer. My fiancé, Heidi, is a cancer survivor. She was accidentally diagnosed five years ago with a stage I non-small-cell adenocarcinoma tumor in her lung. “Accidentally” because she went in for a chest x-ray because she had some pretty serious bronchitis. When she woke up from the surgery which was to remove the newly-found tumor, she was told they’d removed her lower left lobe, which equates to a significant part of her lung capacity. She was a tad dismayed, but when she was told that she was clear of cancer—cured, as it were—she went on her merry way.

…until somebody reviewed what they found in her lung and decided that the lymph system had been involved. And so they threw the kitchen sink at her: chemotherapy with cisplatin. Having survived that with flying colors, she was declared cured pending five years of followup.

In the meantime, she and I met in October of last year. Because of COVID Spring 2020, we had considerable time together, learning more about each other than I think either one of us thought possible in such a short time. She watches out for my health, and I watch out for hers. Neither one of us is a spring chicken, having just turned 50-years-young, but we’re in pretty good shape. She had some bronchitis early this spring with a persistent cough, but that’s about it. And I have an array of weird genetic and non-genetic anomalies that make me just interesting enough to the docs to keep them interested.

And so it would not have been at all surprising to anybody if Heidi had called and declined a 5-year CT scan, saying that COVID-19 “is a thing” and that she feels fine, which she does! But no. Instead, the ever-compliant patient dutifully reported for her final and fifth annual CT scan, the one that decides whether she has earned the label “cured.” A few zaps of the X-rays later, and she was prepared to hear the words,

“Off you go! You’re cured.” Whee! It’s all down hill from here!

What she heard devastated her and me both. Instead of “clear” and “unremarkable,” the words were “concerning” and “suspicious.” Her oncologist called her in and explained that there appeared to be cancer in several lymph nodes. Worse yet, if it really were metastatic non-small-cell lung cancer (mNSCLC), then the treatment plan would be palliative.

Palliative Care is not End-of-Life Care (Even Though it Kinda’ is…)

For those of you not familiar with what palliative care is, let me explain that it’s not the same thing as comfort care, such as is hospice care. What it means is that the treatment is not going to cure the patient and is going to be something that lasts throughout the rest of the patient’s life. But because cancer is a mean and nasty disease, palliative care would last for months, maybe 12-18. And “years” is out of the question.

There was a lot of crying that day. I’m a big man, and yet she and I cried on the floor of her bedroom. The sobs came easily, because why wouldn’t they? Also, I thought palliative care was the same as hospice care, so I was really angry at God. We had met our ideal partners. We’d just started thinking about what retirement might look like. We had committed to each other, regardless of our marital status. Our families had adopted us into the other’s family with such rapidity that we were both amazed. Life was just getting started. Yes, even at 50. This was a blow.

What I didn’t know about her is just how feisty and fierce she really is. At 5’2” (rounding up these days), she is a tiny human, as she likes to say. “Small but mighty!” are her watchwords. And so she dragged me upward out of the anxiety and depression that I sank into. What we did worked well for us; maybe it will help you, too, whether you’re the caregiver or the cared-for.

Mentally Dealing with the Diagnosis

Any time I asked, “How are you feeling?” she would respond with, “I feel well. I’m not anxious. I’m not worried. I’m a little annoyed.” and I would repeat each phrase back to her, “You feel well. You’re not anxious.” and so forth. It became a mantra. It was edited a little bit as time went on. She stayed annoyed for some time because, really? Returning lung cancer? After five years? For a never-smoker? Who wouldn’t be annoyed?!

To this day, we continue this practice. Sometimes we add, “And we’ll be OK.” We don’t really know that in the earthly sense, of course, but we have it on good authority that the long term—eternal, that is—sees us as being more than OK.

Denial Doesn’t Usually Work

The next step, of course, was flat out denial. Well, these are lymph nodes, after all, and she was sick, so isn’t it possible that it’s an infection and the lymph nodes are doing what they’re supposed to do? Her oncologist told us “No,” and the cardio-thoracic surgeon who was consulting with us at the time, and who had performed the lobectomy years before, spotted us a “Well, maybe, but highly unlikely.” The oncologist ordered a PET scan. She reported for that PET scan and we anxiously awaited the results.

In the meantime, the oncologist had ordered a stat biopsy of one of the nodes and an MRI of the brain, but we didn’t know about the MRI yet. Let me tell you on my good authority, you don’t screw around with the emotions of a cancer patient. Here’s how I know.

Don’t Mess with a Cancer Patient’s Emotions

We had arranged a family vacation to Pawley’s Island, South Carolina, right in the middle of a COVID-19 hotspot, for late July to celebrate Heidi’s 50th birthday. My parents, my sister and her family, and the two of us would be there. While we were waiting for the rooms to be ready, a scheduler called from the imaging center (not Hartford Hospital’s, but one that will remain nameless) and asked Heidi when she would like to schedule her stat MRI of the brain and a biopsy to be performed in four to six weeks.

When it’s the scheduler telling you that an MRI to detect brain tumors has been ordered (but not mentioned by the oncology team) and she can’t explain anything, and when it’s the scheduler telling you that your stat procedure won’t happen for a month or so, you tend to explode, but in the nicest way possible. You tend to ask why these orders contradict the ones you know about. You get very sarcastic and cutting. You begin tearing the scheduler “a new one”—in a nice way. You then begin to worry about why the oncologist has ordered an MRI of your head because isn’t the cancer in your lymph nodes in your chest and just what the hell is he looking for in your head that you don’t know about?

At that point, through anger and tears, you call your oncologist who can’t be reached because it seems he’s on vacation again (well-earned—we get it!) and the oncologist’s MA turns out to be the most helpful person in the world. Kudos to Kristan; she figured out what was going on and to make a long story short, the scheduler had no idea what she was talking about, the scheduler wasn’t supposed to have called the patient at all, and Kristan would have been able to help if things had gone right in the first place.

All was figured out when the MRI was explained off as standard procedure (though we didn’t understand until later) and the biopsy was scheduled for the day after we returned from vacation.

During our vacation, we prayed hard for an infection. Heck, we even prayed hard for COVID-19. Anything but cancer.

And during our vacation, the cardio-thoracic surgeon called and walked us through the results of the PET scan.

It was cancer. No two ways about it; it wasn’t an infection. It wasn’t COVID-19. It wasn’t an erroneous result for somebody named “Peidi Hartain.” It was Cancer. Again.

We cried that day, too, and we prayed for a miracle.

Why an MRI is Needed Even with a PET Scan

That didn’t necessarily make us feel better, but his explanation to us why the MRI of the brain was necessary did make us feel better. The PET scan shows the cells in the body with high metabolic rates. Cancer cells, which eat more food (if you will) and grow faster than other cells in the body, have high metabolic rates. So they show up as bright colors on the PET scan. The other cells that consume a lot more food in the body are the brain cells. The brain lights up like an orange Christmas tree! A ball of sun! A bright light! And now I completely understand why depression, which kicks the brain into overdrive, can cause weight loss! (It normally accounts for about 20% of the body’s energy use.) Looking for an orange tumor cell amongst all the orange brain cells is hopeless, like staring at the sun to see sunspots. (Don’t try that.) And so an MRI is needed to look for masses that are non-brain, e.g., cancerous, in the head.

However, we had some hope left, and we now hung our hats on the glimmer of hope that we had. Because of some other lab numbers that Heidi showed, the cardio-thoracic doc said it could be run-of-the-mill lymphoma, something that is somewhat manageable and has a pretty good prognosis with a lot of folks living five years or more. That’s still not great, but any time somebody tells you that you might have metastatic lung cancer, your assumption is that you’re going to die very much sooner than five years. (And you’d be right for the most part.) The only way to find out what it really is was to do a biopsy, and so a bronchial biopsy would be performed under general anesthesia unless… unless what? I truly don’t remember now because that was about ten years ago and I can’t remember what caused a deviation from that plan.

In any case, a needle biopsy ended up being ordered on one of the axilary lymph nodes/tumors. And so a day after we returned from Pawley’s Island, paradise of the Southeastern Seaboard, Heidi was in an exam room having a needle poked around inside her armpit while her sister and I quaffed a beer at a nearby restaurant.

Never Underestimate the Value of Family

She wasn’t my sister-in-law yet, but she was there for us both. She’d flown up that week to support Heidi and, by extension, me, through some of these more difficult events. Her presence was truly a godsend. Now, if your relatives come and all they do is sponge off you and lie about on the couch all day, maybe having your family come to support you isn’t the best idea. Having Heather here, however, was helluva good. She was calm, pragmatic, and helped us avoid the emotional pitfalls we ran into.

Pretty soon, we got those results, and the answer really was, as her oncologist had expected, mNSCLC. However, the biopsy would be sent out for what is known as “moleculars,” which is a shorthand way of saying “molecular diagnostics.” The moleculars examine the genetic materials of the sample to see if there’s something wrong with them that can be treated. The oncologist told us that day that the previous tumor was PD-L1 and ALK negative. Dammit. There go the most popular treatments including immunotherapy targeted at the PD-L1 gene. On the other hand, PD-L1 is present in more-aggressive tumors (as we learned from the Interwebs and reading many papers online), so maybe that’s not such a bad thing after all.

That day was not a good day for the people at the physicians group. Why? Here’s what happened:

We visited the oncology clinic after confirming via E-mail that the appointment would be face-to-face and not telemedicine. We were told, rather rudely, that the visit was supposed to have been telemedicine. The receptionist did confirm with Heidi’s oncologist (to be fair, a wonderful doctor and caregiver), that he would see her and announced to Heidi that “The doctor will see you after all.” It was a rude start to the visit.

The nurse who checked Heidi in was pleasant, and for that we are grateful.

However, after the visit in which the diagnosis was pronounced, the doctor took us to the checkout desk where the scheduler was clearly annoyed to have had her lunch interrupted. We were not greeted in any way. She finished what she was eating. We stood there as something happened on the computer. Still, no greeting or word from her. When prompted to set up a port placement, she was unable to answer Heidi’s questions in a way which did not cause Heidi frustration. We left that clinic with a most dissatisfied opinion of its clerical staff.

Clearly, the staff needed training in patient compassion. This clinic does not deal with runny noses all day; they deal with patients whose diagnoses are life-altering and life-ending. Their attitude should match the nature of their clientele. And if they can’t be bothered to interrupt their lunch to deal with someone who just had a Stage IV cancer diagnosis pronounced, they should have been fired on the spot.

Again, there were tears shed. There were prayers said. And as Heather has oft said, “Heidi, you can’t die yet. You haven’t pissed off the last person you’re supposed to piss off.” Ain’t over ‘til the small lady rings (and presumably pisses you off during the call).

During the visit, a treatment plan was put into place that matched what we had expected, namely chemotherapy. What we were surprised by was that the standard of care for mNSCLC included an immunotherapy drug: Alimta (a chemotherapy drug), carboplatin (a chemotherapy drug) and Keytruda (an immunotherapy drug targeted at the PD-L1 genetic mutation). That last one was the surprise, precisely because of its target, which Heidi doesn’t exhibit. But it helps—the trials say so, and, well, that’s good enough for us.

In the meantime, we got in touch with the Dana-Farber Cancer Institute, with our oncologist’s blessing, because if there’s something in the works, in a trial, etc., we wanted to know about it.

Thinking that we’d have the results of the moleculars back by two weeks after the biopsy (which sounded reasonable to everybody), we scheduled an appointment with DFCI and one of their Enormous Brains. (And by “we” throughout, I really mean “Heidi,” because she is super at taking care of these sorts of things. It’s a large part of her life’s work for BrightStar Care of Hartford, her company.) So we ventured north to Boston to visit the Enormous Brain.

Without the moleculars.

Yep, the moleculars were nowhere to be seen. But Dr. EB, a research oncologist specializing in thoracic cancers and who sees patients 50% of the time and researches 50% of the time, spent a good hour educating us about mNSCLC and his thoughts on the origin of this recurrence. His thoughts: for a non-smoker, a lung cancer well-removed, and over-the-top treatment of chemotherapy which was overkill to say the least, it had to be something genetic. He recommended that we go ahead and do a blood test that would reveal if there were any mutant genetic material floating around in the bloodstream. These random bits of genetic material get tossed into the bloodstream because aggresive, rapidly-growing cancer cells are, in my words, careless and messy, throwing DNA and RNA around and making a mess of otherwise-clean blood. The results of that test would take only 7-10 days; we weren’t sure how long it would take for the moleculars to return. It just made good sense.

We learned a considerable amount about targeted therapies at this meeting and it was clear that we should be really hoping for a genetic mutation that is treatable with a targeted therapy. Chemotherapy is a non-specific therapy and targets all cells which are fast-growing. That’s the cancer cells, but it’s also hair cells and lots of other cells that get killed in the crossfire. (Why it doesn’t kill brain cells, I have no idea.) Targeted therapies, on the other hand, work to attach themselves to the genetic mutation and somehow kill the cell or otherwise prevent its reproduction. They have some side effects, too, but they aren’t nearly as destructive as chemo.

To use Keytruda, or not to use Keytruda: That is the Question

Dr. EB also recommended that without the moleculars, we not proceed with the Keytruda for the first cycle of chemo. Keytruda makes lung infections more likely, and some targeted therapies also make lung infections more likely. Because Keytruda has a half-life of 26 days (meaning it’s in the body in large amounts for a considerably long period of time, with 25% remaining at the 52-day point and about 6% at the 104-day point) and because nobody’s sure if the risk of lung infection is additive or exponential, he said that if we used Keytruda, it might prevent the use of a targeted treatment if one were available. 

We agreed. He is Dr. EB, after all.

(To be fair to our local oncologist, I’m sure he’d reached the same conclusion. He’s a smart cookie.)

The best news of all, however inconsequential to her physical health it may have been, was that the three-drug soup prescribed is not so caustic as to require a port placement. What’s a port? A port is basically a piece of plastic that goes under the skin near the upper left of the chest, right about where your fingers rest when you say the Pledge of Allegiance (provided you’re not kneeling). It connects to the vena cava, a very large blood vessel that has lots of room for caustic drugs to dilute before really getting up against other tissues. When you go in for chemo, they poke a needle through your skin and into the port and deliver the drugs through the port into the big pipe. It’s a big convenience for the nurses and protects your veins and skin from the deadly (literally) drugs you’re receiving.

It’s also an emotional hurdle if you had one five years ago and fought to have it removed. It’s a symbol of the cancer’s return. It’s a touchpoint. It’s something you will fight to avoid if you can. And when you’re told that you don’t need a port, you rejoice if your name is Heidi. And that’s consequential to your mental health.

We wandered down to the phlebotomy lab and, as elsewhere, the people were kind and compassionate and willing to explain everything about everything, education seemingly part of their mission. A very nice lady poked Heidi’s arm nearly painlessly and drew the blood to be sent off to California for the testing. After that, we left for home.

Our trip home was subdued because we didn’t learn anything that was particularly promising. On the other hand, we hadn’t learned anything that was negative, either. We gained a little bit of hope in the blood test, learned about genetics and testing and targeted treatment. We also learned that Dana-Farber is a superior organization in every way to any we’ve encountered yet. That’s not saying anything about most of the individual people in the organizations we’ve encountered, but as a whole, these organizations pale compared to Dana-Farber.

Why I Would Go to Dana-Farber Cancer Institute Even if I had to Travel a Long Way

It was clear from the moment we were greeted by the parking attendant to the moment we left and were greeted by the other parking attendant that everyone at Dana-Farber was thoroughly aware that the patient and her family were there because they have a very serious illness, one which is causing mental stress, anxiety, fear, and other emotions which make coping with the myriad details of a complex visit difficult. They made every effort to make our visit as smooth as possible. Each person was efficient and effective in their roles, and yet each person we encountered was pleasant. As we left, we recognized that the pervasive atmosphere was compassion. Every single person we encountered was compassionate toward us. Though the news was no better than we had heard elsewhere, the staff around the doctor at Dana-Farber exhibited respect for the condition of the patient… and showed compassion.

Will the Waiting Never Cease?

After our visit with Dana-Farber, we consulted again with the local oncology group and set up a port-free chemo session. They agreed with Dr. EB that Keytruda would not be given for the first round and most certainly we’d have the molecular studies back sometime soon to determine what to do with the second round. They also agreed with him that a CT would be useful to assess the starting condition at the time of first round of chemo. That would give us a useful reference point to compare to when treatment is done.

Shortly after we spoke to Dr. EB the first time and had the blood draw, we had another call with him to review the results of the blood analysis. It showed almost nothing, with “nothing” being a distinct possibility. Remember that it takes an aggressive cancer to throw off enough genetic material to test? That they didn’t have enough to test is good news as far as we’re concerned. However, the results did say that there was possibly an RNA mutation, but the amount of material was so low that it was just below their threshold to call it a finding. So that report showed, essentially, nothing. And for the reason above, we considered that a good thing.

The local oncology team and Dr. EB are working well together. I do not mean to insult the local folks, either: they are danged smart and I make no comparison about their abilities to the folks at Dana-Farber. They’ve had to call the lab repeatedly to prod forward progress on the moleculars. They’ve had to answer the numerous calls we’ve made to figure out what next steps should be in light of changing circumstances. They’ve done a yeoman’s work in keeping Heidi on the right track. And it is clear that they do care. (I wish I could say the same about their support staff.)

However, the moleculars from the lab never seemed to be headed to us. It was now about five weeks after the biopsy was done, and everybody seemed to think that the results should be back. And yet we waited.

Meanwhile, the first round of chemo was administered without a hitch without a port, and Heidi was happy! Of course, after the steroids wore off, she crashed hard and was exhausted for about three days. We caught up with the oncology APRN the following week and listed off all the symptoms and were told, Yep, that’s normal. (Whew.) Things, overall, had gone pretty well.

The Waiting Ceases

We finally got the report back for the moleculars that day, too, six weeks after the sample was sent to the lab. However, they didn’t look all that great. Of the 44 DNA mutations tested for, she had only one, a PTEN mutation—and there’s no targeted treatment for that… yet. It’s an interesting mutation in that it seems to be present in a bunch of different cancers (including mNSCLC) and is a cancer suppressor. If yours is off, you’re more likely to have cancer cells running amok. And it’s hereditary. (Oh, dear, kids…)

Good news, though: initial studies have shown that cruciform veggies (e.g., broccoli, cauliflower, Brussels sprouts) have a compound which is abbreviated I3C that can, through a molecular pathway, turn that gene back on. Of course, to be effective, you’d have to eat six pounds of raw Brussels sprouts daily, so that’s not a likely solution. (There is an OTC supplement, but don’t go out and try it without talking to your doctor, of course! We asked Dr. EB about it and the problem with all of the studies so far is that they’re not in clinical trials, nobody knows whether it will help or harm tumor growth, and it might interact badly with the treatments you’re already on. So, Brussels sprouts in normal quantities, lightly sautéed in butter, please.)

Once we got that report back, we discussed it with the local oncology team and had it sent to Dr. EB with whom we had a third discussion the day before our wedding. (Yes, we got married. Heidi says I’m either brave or crazy. I just say that nobody ever talks about being sane in love and leave it at that.) During that conversation, we pointed out that the RNA results were strangely still pending. We also pointed out that the blood test showed a possible ROS1 mutation… which is an RNA mutation! In other words, there was a good possibly that there was a ROS1 mutation still out there but that the outstanding RNA tests didn’t show it yet. He was hopeful and curious in a very genuine way.

We held off on the port placement again (Heidi just couldn’t emotionally handle the port as a last resort yet) and forged ahead towards chemo round two. One week after our wedding, the local oncologist called and said that he had pestered somebody into giving him a preliminary result from the biopsy, eight weeks after the core sample. And guess what showed up?! Yep! She has a ROS1 mutation in her RNA! Yay! That’s one of the targeted treatable RNA mutations—of which there are only four tested for! He sounded genuinely pleased and happy for us because it is a big deal. The treatments available show an efficacy rate of 70% at slowing or shrinking tumor growth. That’s twice as good as chemotherapy, and some people live for years on these drugs.

Some.

But not all.

However, it’s still wonderful news. His recommendation was to continue with the second round of chemo because it would take a week to get the written report from the lab and then another week to get the drug approved by Heidi’s insurance. In the meantime, two weeks after when she would have round two administered would have passed, and leaving the cancer to do what it wants to for two weeks is not a good idea.

We called Dr. EB and he agreed with everything that the local oncologist said. His one hesitation was about round two, but after we explained why there was a two week delay, he agreed. His one suggestion was to get a second CT scan at the end of the 21 days following the second round so we know what the effect of the chemo is independent of the targeted treatment. At some point, it’s likely that the cancer will resist the targeted treatment, and it’s possible that Heidi’s in the 30%, so having an idea of the efficacy of the chemo is a good idea.

And so here we are. Heidi is ROS1 positive (ROS1+). And we’re learning all kinds of stuff about ROS1 fusions—it’s not really a mutation because some of the curly bits of the RNA have fused together at the ROS1 spot (i.e., normal RNA, just fused together where it shouldn’t be). And the more we learn, the more we can’t possibly believe that all of this stuff works and is self-repairing and so forth without a stronger belief in God. Now, we choose to believe in the God of the Bible. You may choose to believe in intelligent design. In either case, we both agree that this stuff is just too complicated to be random. It’s insanely complex. And yet… it… works….

We learned about a ROS1 treatment with a drug called crizotinib, the brand name of which is Xalkori. (I commented to Dr. EB that the generic names seem to be tongue twisters. He thinks it’s to make it more likely that we’ll call it by the brand name. I tend to agree.) Xalkori a Pfizer drug. And we will be investing in Pfizer because the daily dose is 2 capsules. At $320 each. A month’s supply is about $19,000.

Nineteen. Thousand. Dollars.

But good news! There are coupons available! (Not kidding.)

So remember we prayed for a miracle? No, we’ve not seen if this drug is effective or not, but ROS1+ is a 1% mutation of all non small cell lung cancers. So Heidi is one of about only 1,700 people annually for whom this drug would be useful. Wonder why it’s so high priced? (It’s Pfizer’s third-highest grossing drug.) Me too, but I have a good understanding of supply and demand, the economics of patents and research, and I’m not surprised that it’s expensive.

I am a little taken aback that it’s that expensive.

But we are both blessed with health insurance. And we have been blessed with each other and the support we’ve been able to give each other.

Thanks be to God.

We’ll continue to update my blog as we progress. I know this is a big brain dump, and you can see that it’s a bit of an emotional roller coaster. There is much left to come.

Resources

If you are a lung cancer patient, find a support group.

If you’re a caregiver, find a support group, too. It’s an emotional road. You’ll need your own Heather.

And donate to the American Lung Association. Please.

Lastly, if you’re ROS1 positive, check out the ROS1ders. This is a small group of people who are distilling a lot of deep science and drug info into regularly-updated summaries. Support them, too, with your time and talents.

Continue to Part 2

[Edited 9/11/2020 to clean up some punctuation, extra-long sentences, spelling (though I’m sure I’ve missed some), and such. WNE]

A few evenings ago, I was the most frustrated person on the face of the planet. I can tell you this with some assurance because I have a meter on my Dashboard that tells me so. It measures my typing pace and the frequency of command line errors and... I digress.

The root cause of my frustrations was a file named #org.clamav.freshclam.plist# (whose origins I'm still unclear on, though I assume it's a leftover emacs temporary file or something). Every time the system booted, launchd tried to load that plist file, located in the directory /System/Library/LaunchDaemons/, and the net result was that postfix wouldn't start because amavisd wouldn't start. Why amavisd wouldn't start, I still don't know, because it seems to me it shouldn't depend on freshclam to run. But I couldn't tell what was going on by looking at any of the logs because there is not enough information shown.

And therein lies the heart of the matter: launchd, though it doesn't operate quite silently, doesn't have a debug mode. Furthermore, even in normal operation, it doesn't quite give enough information to make for productive bug hunting. I thought of these problems after the frustration level quieted down the following morning when I figured out what was going on.

First, What was going on?

What I knew was that postfix wasn't launching and neither was amavisd. I also knew from these entries in the system log

Oct 8 19:26:03 shr-g5 org.clamav.freshclam[3087]: ERROR: Incorrect argument format for option <strike>-checks (</strike>c) Oct 8 19:26:03 shr-g5 org.clamav.freshclam[3087]: ERROR: Can't parse command line options Oct 8 19:26:03 shr-g5 com.apple.launchd[1] (org.clamav.freshclam[3087]): Exited with exit code: 1 Otc 8 19:26:03 shr-g5 com.apple.launchd[1] (org.clamav.freshclam): Throttling respawn: Will start in 10 seconds that there was a problem with the file /System/Library/LaunchDaemons/org.clamav.freshclam.plist (or so I thought). I have mentioned this problem before, as well as the fix.

Great! I know how to fix that! So, armed with the confidence similar to that elicited by carrying a howitzer into a rock/paper/scissors match, I edited the file at fault because I had indeed messed around in Server Admin and knew that SA rewrites that file and screws it up again.

So I was perplexed when I kept seeing these errors over and over and over again. Stranger still, I could do a

sudo launchctl unload org.clamav.freshclam.plist followed by a sudo launchctl load org.clamav.freshclam.plist and freshclam was happy! I could then manually launch amavisd and postfix and life was good, until I rebooted and the same frickin' problem reappeared.

And that left me extremely frustrated, even more so than usual because I had a rented Bobcat-style digger sitting out in the driveway consuming my money without being productive for me at all. Eventually, I threw in the towel and started digging, which turns out to be very therapeutic.

[The paragraph that belongs here was removed because you really don't want to hear me gripe about restoring a MacOS X Server box from a Time Machine backup and all of the problems that that process entails. I've mentioned that before. Suffice it to say, I did so, it worked, but the problem remained.]

The following morning, again, with the digger outside consuming money at an alarming rate, I gave myself until 9am to figure out the problem. After that, my users, a.k.a. family, would have to live without E-mail until the digger was returned around 5pm. Fortunately, I did a ls without my usual -la and saw the aforementioned #'d file. I guessed that it was the cause of all my troubles, moved it out of the way, and the machine booted happily with no launchd errors.

So that leads me to the second part of my treatise, namely, What would have made this experience much less frustrating?

I have three suggestions. The first is that launchd should report the path to and name of the file it's parsing so that I would have seen immediately where the suspect file is. This information is especially important as there are several locations that launchd looks for plist files.

The second is that lauchd should tell me what command it tried that caused the problem. That's just good debugging practice.

Resulting logs would look like this in Bill's World:

Oct 8 19:26:03 shr-g5 org.clamav.freshclam[3087]: ERROR: While processing '/System/Library/LaunchDaemons/#org.clamav.freshclam.plist#' Oct 8 19:26:03 shr-g5 org.clamav.freshclam[3087]: ERROR: Issued command '/usr/bin/freshclam (rest of options from plist file here)' Oct 8 19:26:03 shr-g5 org.clamav.freshclam[3087]: ERROR: Incorrect argument format for option <strike>-checks (</strike>c) Oct 8 19:26:03 shr-g5 org.clamav.freshclam[3087]: ERROR: Can't parse command line options Oct 8 19:26:03 shr-g5 com.apple.launchd[1] (org.clamav.freshclam[3087]): Exited with exit code: 1

The third thing I'd like to see is more info output by launchctl list. As it is, it doesn't show any info about what's loaded except what the base name of the plist file is. No information about the issued command, no information about the path to the plist file, names of plist files which were tried and found wanting, nothing else is available that would have made this debugging process any easier. And that would be helpful, too.

launchd is a really cool new thing, but it lacks some of the polish and usability features that the more mature things have in place already. These are my suggestions which, I think, will help improve launchd and make it a cool, mature thing.

In my original posting, I gripe that Time Capsule makes a royal mess of things for a MacOS X Server box. And it does, still, unless you do some system file editing.

But I’ve had a few thoughts since then, especially since I just went through this whole scenario again last night. This time, I was better prepared, though, as the system file editing I describe in that article saved my bacon.

Or my users’ IMAP data. Bacon, IMAP data… if only IMAP data tasted as good… mmm… bacon…

Sorry. Back to reality.

Apple has chosen to keep several /var directories out of the Time Machine backups because they change rapidly, there’s a lot of data, and it would fill up the backup disk PDQ. Wisely, they also include some preferences in the exclusions file to take care of this little problem, namely the keys “PathsExcluded”, “ContentsExcluded”, and “FileContentsExcluded”.

Entire directories can be excluded with the first, the contents of directories can be excluded with the second (i.e., it preserves the top directory structure, but doesn’t backup any files or subdirectories), and the third backs up all of the subdirectory structures, but still no files. Googing these three terms yields nothing, so I assume I have a correct understanding of what they do.

If I do understand these correctly, then there’s a fourth kind that needs to be created, namely something which means “backup only the file permissions, names, ACLs, etc., but don’t backup the data in the files.” This key would allow the /var/log directory backup to maintain zero-length backup files (hence, they never change) but allow not-so-smart software which doesn’t/can’t create its own logfiles when missing (Apache, I’m lookin’ at you) to use them when restored from backup.

Maybe one of these keys means exactly that, but I’ll be darned if I know which one, and, like I said, there doesn’t seem to be any documentation on it.

I have a 1TB Apple Time Capsule which I say by way of introduction and not as a means to lord over you, dear reader, that I’ve got something that you might not have. I use this Time Capsule to backup my MacOS X Server.

According to the product page, here’s what Time Capsule can do:

Time Capsule is a revolutionary backup device that works wirelessly with Time Machine in Mac OS X Leopard. It automatically backs up everything, so you no longer have to worry about losing your digital life.

and

Backing up is something we all know we should do, but often don’t. And while disaster is a great motivator, now it doesn’t have to be. Because with Time Capsule, the nagging need to back up has been replaced by automatic, constant protection. And even better, it all happens wirelessly, saving everything important, including your sanity.

The only requirement for your Mac is that it is running MacOS X 10.5.2 or later.

That’s because MacOS X 10.5 introduces Time Machine like this:

For the initial backup, Time Machine copies the entire contents of the computer to your backup drive. It copies every file exactly (without compression), skipping caches and other files that aren’t required to restore your Mac to its original state.

and

By default, Time Machine backs up everything on your Mac.

(Emphasis mine.)

So when I hooked it up to my MacOS X Server 10.5.4 server and turned on Time Machine (one click!… and some typing to set the password, but we’ll overlook that), I was pleased to see that, indeed, my server was backed up. I could browse the backup and see it was good. Cool!

But then the other day, disaster struck only hours before we were to leave for the airport: my server had crashed in such a way as to be unrecoverable. Things were grim, and the only thing I could do was to restart from the Server installation DVD and restore the machine from the Time Capsule. Well, it did that in about 9 hours and with some remote system administration from my beautiful wife, the machine was back up and running.

Except it wasn’t.

E-mail didn’t work. Web services wouldn’t start. All hell was breaking loose in the system logs as various services which were trying to start up just plain wouldn’t start up. Things which previously had a quiet existence on this machine were suddenly vociferously complaining about a plethora of problems. While each had its own gripe, most were unhappy about the nonexistence of a log directory, /var/log. “Huh?” I thought to myself, “I thought I read that Time Machine backed up my entire machine to get it back to its original state. What’d I miss? And why do I feel like it’s my fault all of a sudden?”

A little searching on the web reveals that there’s a list of stuff Time Machine doesn’t back up which, on a normal MacOS client machine might be OK, but for a server is disastrous. The list, stored at

/System/Library/CoreServices/backupd.bundle/Contents/Resources/StdExclusions.plist

has, among other things, these items which are excluded

/private/var/log

and

/private/var/spool

OK, log files don’t necessarily need to be backed up, maybe only the last one so you can see what happened before the crash would be nice. Nonetheless, if there are various services which need log files or log directories to exist to run, then something, somewhere must recreate these logfiles or the system never gets up and running, and the backup has, in fact, failed. Apache is quite content to gripe that it can’t make logfiles. Amavis, too, can’t do anything unless the directory is there. Sorting out which logs need to be where and who owns them and what their permissions should be took me the better part of two hours, and I’m know I don’t have them all right. (And that’s only for the few services I’m running. God only knows what I’m missing for the others.)

But… and this one is inexcusable… not backing up /var/spool, which includes /var/spool/imap which is where my IMAP users’ E-mail is stored!! is insane and has my blood boiling. This is an oversight which is completely uncharacteristic of Apple but for which there is no excuse.

The next four hours I spent trying to recover my IMAP users and getting Postfix to run were maddening. I had lost nearly a gigabyte of E-mail. Multiple IMAP directories had to be “reconstructed,” sometimes a success and sometimes not, according to the many webpages out there. Even Apple’s own webpage describing this process failed. (Something about partition “/var/spool/imap/user” not existing.) My users were similarly peeved. “You mean we bought that expensive Time Capsule instead of a simple external hard drive for mirroring and it didn’t work?!”

Never mind the fact that it’s not the Time Capsule at fault, it’s Time Machine. They didn’t see it that way. They saw an expenditure that was unjustified because it simply didn’t work. I saw a maddening amount of work on a weekend, on my vacation, from 1000 miles away, because it simply didn’t work. And that’s just wrong, wrong, wrong.

Note to Steve Jobs You would be incensed, too.

My users and I are angry, and rightfully so. You, Apple, make a promise about a very important function and you don’t keep it. This is backup, for goodness’ sake! This is the kind of thing that has to work. If you say “backup,” it implies that it will, it shall, it must work.

And it didn’t.

Another note to Steve Jobs: If this had happened to you, you would have seen to it that somebody got yelled at and it would have been fixed immediately. Really. This is the kind of thing you hate. (I’ve read enough of Fake Steve Jobs to know how you think, man.)

Coming in an article “real soon now:” an article about how to recover from the various messes left behind by a Time Machine restore of MacOS X Server. (Just as soon as someone answers the question I pose here, that is.)