Skip to content

NLP and The Wheel Of Time – Part 1

I love epic fantasy stories. All the characters, plot lines, world-building, and the little details that foreshadow big developments — I devour them in multiple readings. I also have some time to play with the Google Cloud Platform’s Natural Language API. So I decided to see what analysis I could do with that API on one of the most voluminous and detailed epic fantasies I’ve read: Robert Jordan’s The Wheel Of Time series.

I’m going to blog my efforts, in case something interesting or useful shakes out. I’m using this opportunity to teach myself Python as I go, since that language is popular in the A.I. work I’ve seen. So there will probably be some discoveries (and horrendous examples of code) along the way.

My rough plan has the following milestones:

  1. Get the Wheel Of Time books in plain text format, so the Google APIs can read them.
  2. See what the API’s sentiment analysis, entity analysis, entity-sentiment analysis, and syntactic analysis data looks like for The Wheel Of Time.
  3. Use sentiment analysis to graph the emotional arcs, in total and maybe of characters, of the story, to compare them to the “six main story arcs” discussed in this article in The Atlantic.

After that? Let’s see where the data can take me. I have some thoughts on creating a system that can answer questions about the story, and possibly expanding the training model to include labels and concepts, but I’ll focus on my first three milestones to begin with.

Part 1: Get The Wheel Of Time In Plain Text

Tor Books has the commendable policy of selling all their eBooks unlocked and DRM-free, and I already have the books on my Barnes & Noble Nook, so I started with the EPUB file format.

Unfortunately, the Nook app on Android devices hides your eBook files in a directory you can only access if you have root access to your device. I wasn’t interested in the warranty implications of going down that route. But the family Windows 10 machine has a free Nook app that downloads your eBooks. After that, it’s just a matter of searching the drive for where it put *.epub. I found them in the rather obscure directory:

C:\Users\[myuser]\AppData\Local\Packages\BarnesNoble.Nook_[somehash]\LocalState

Huzzah! Then I renamed the file as a .zip (an .epub is a .zip with a particular directory structure), and dug into the ZIP archive. Unzipped, the file looked like this:

+- META-INF
|
+- OEBPS
|    +- Images
+- mimetype

The files I’m interested in are in OEBPS\. Each chapter or section (basically, each table of contents entry has an HTML file, conveniently named for sorting by the chapter number. The markup is clean and well-formed, and the style classes are intuitive. Cleaning it would be straightforward.

Now I had to learn some Python. I was familiar with the syntax, and I was an experienced Java programmer, so most of what I had to learn could be found on StackOverflow. Unless I got fancy, it would be a 1-use script, but there are 13 books in The Wheel Of Time (including the prequel, New Spring), so it wouldn’t hurt to take a stab at maintainability. I wanted to do proper Test-Driven Development, but I was getting impatient to see progress, so I just kept tweaking-and-running until I got it to work on real data.

I envisioned three components:

  1. Something that stripped out HTML markup from a character stream and left readable plain text.
  2. Something that created a plain text file from an HTML chapter file, using the first component.
  3. Something that took an EPUB file, unzipped it, iterated through the chapter files, used the previous component to make text file equivalents, and zipped them up for transfer to Google Cloud Platform, or wherever.

Python had a built in html.parser.HTMLParser which did exactly what I needed, for the first component. After that, it was all file I/O and some ZIP manipulation, all with standard packages. The “if __name__ == “__main__” construction for an executable class seemed awkward, but otherwise I was impressed with how compact the code was.

If you want to see my ugly-but-functional beginner Python code, feel free to peek on Github.

At this point, I had The Wheel Of Time all in a ZIP of plain text files. I was ready to figure out how to use the Google Cloud Platform’s Natural Language API. That will be in my next post.

 

Might As Well Jump

 

 

 

In Frank Herbert’s Dune, the mystical Bene Gesserit sisterhood had a litany against fear.

“I must not fear.

Fear is the mind-killer.

Fear is the little-death that brings total obliteration.

I will face my fear.

I will permit it to pass over me and through me.

And when it has gone past I will turn the inner eye to see its path.

Where the fear has gone there will be nothing. Only I will remain.”

To the Bene Gesserit, what separated a human from an animal was the ability to withstand and endure terror, suborning it to one’s will. The animal reacted. The human planned.

I’ve always been a planner. I got afraid if I didn’t have a plan. So when people ask why I left my job without another in hand, I tell them “I was about to turn 46, with more years behind than ahead, and my ‘maybe someday’ list was running out of somedays.” When it’s more than a passing conversation, I quote Annie Dillard in The Writing Life: “How we spend our days is, of course, how we spend our lives. What we do with this hour and that one is what we are doing.” They commend me for having the courage to pursue my dreams, which embarrasses me because it had nothing to do with courage, or even dreams. I’m privileged to have resources to get by while I figure things out. My family is in no real danger. All I did was react to my calculus of competing fears.

They weren’t even interesting fears. On one side was the mid-life crisis of a plateaued career, mounting frustration, and a nebulous sense of being trapped. On the other were the worst-case scenarios of starting over. The boredom I knew, versus all the worse ways I could fail. Endurance versus action. I flipped the Bene Gesserit’s script, building up a fear-driven resilience to the daily grind that paid the bills and gave me a respectable job title. I could detach my emotions, do the work, and set my brain on autopilot. “The examined life is not worth living,” I’d joke. Because once examined, I’d have to respond to what I saw. And that meant I’d have to leap off the edge of that plateau, into the unknown.

Jumping was on my mind when my supervisor took me to task for not being enthusiastic and inspiring enough for my team.

In my sophomore year of college, I joined some dorm mates and went skydiving. It was a half day process of getting familiar with the equipment, practicing the sequence of actions that would get me to the ground intact, and drilling on the safety contingencies. That done, we went up in a tiny Cessna. A static line was attached from the plane to our parachutes; they would open automatically when we fell a certain distance, provided we arched our bodies in the way that kept our backs up and bellies down. One by one, we followed the steps we’d practiced.

When my turn came, our instructor called ONE, and I stood in the open door frame in the side of the plane.

TWO. I stepped out between a pair of bars under the wing; one that I could grip in my hands, and the other that supported my feet.

THREE. While holding on to the upper bar, I stepped off the lower, and hung from my hands under the wing.

FOUR. There was no getting back into the plane now, even if the shame of doing so was endurable. The only things I could do were continue to hang there, or do what I’d come to do, and just. let. go.

Confusion. The roar of wind in my ears through my helmet. A jumble of images and no idea which way was up. Then the parachute opened, and it all went quiet. Peaceful. Free. I hung in the sky supported by air, and drifted over green and brown rectangles of land. I saw a barn. A house. A Christmas tree. It was a sublime, singular experience. Only later did it hit me that I’d jumped out of an airplane. It seemed insane. They key was that I hadn’t approached it as a single action. Even at the very end, it was four steps, which I concentrated on executing as well as I could, my mind focused on that specific task and not the entire chain of events that would put me in freefall, 12,000 feet up.

Of course, underneath it all was a huge amount of faith in the entire support system. I had faith in the instructors, the pilot, the equipment handlers, the equipment itself. I had faith in the invulnerability that came with being 19 years old. More than two decades later, I had forgotten that faith, especially in myself. As my responsibilities expanded beyond myself, so did my caution. I allowed my roles to define me, and as they did, my life became less about me and more about my obligations. That’s even considered virtuous to some. They call it a work ethic. I suspect that’s what a mid-life crisis is: after years of paying dues and being responsible, the sidelined self roars back into focus, demanding satisfaction. It wants, selfishly, unapologetically. In my case, it wanted, once again, to jump.

ONE. I told my supervisor that if my attitude was such a concern, but there was nothing to do about what caused that attitude, I really had to think about why I was here.

TWO. I asked my wife how concerned she’d be if I outright quit. This week. Tomorrow. She had once quit in frustration. She said we could make it work.

THREE. I backed up and cleared away everything personal on my laptop and desk, and took a final inventory of what I’d be leaving behind. Surprisingly little, all told.

FOUR. I looked forward to the follow-up meeting with my supervisor. For the first time in a long time, I felt like I had power. I was no longer beholden, no longer obligated. A feeling of magnanimity swelled up in me. I didn’t need to vent my grievances, I simply thanked him for his candor and give my two weeks notice.

I was light, free, floating and smiling all those two weeks. My relationship to people and objects in and around the office had shifted. I moved among them, but untethered. All the weight was lifted. Ironically, I could focus better, and the magnanimous attitude buoyed me through the remaining work days. I did ask myself if I could have somehow manufactured this way of being while keeping my job, but it was a passing thought. The feeling of release, of hope for the future, couldn’t be faked. It could only come from facing down my fear, letting go, and tumbling into open air.

CODA. The following week after my last day, my phone buzzed with texts, emails, and instant messages. Several people, including my supervisor, his supervisor, and half my peers, had their jobs terminated. A part of me wished I had held out three days, for a severance package. But the better part was happy to have my instincts validated. It’s better to jump than to be pushed.

 

i like my brain unemployed

(With apologies to e e  cummings)

i like my brain when it is unem

ployed. It is so quite a new thing.

Muscles unclenched and nerves calm.

unemployment. i like what it does,

i like its whats. i like to feel curiosity

pull my mind in directions it chooses

unfettered by needs not my own

de foris, from outside, against which i will

again and again and again

chafe, i chafe on corporate culture,

i like the quiet oasis of home, books, articles

and the wonderful terror of the blank page

with pen or cursor hovering… poised

and possibly i like the thrill

of authoring the story of myself

A Wrap-Up From My “Providence” Reading Guide

After twelve issues – or eighteen, if you count The Courtyard and Neonomicon – we come to the end of Alan Moore and Jacen Burrows’ Providence. To finally have the whole project complete at last, we can finally sit back and reflect. No more annotations, no more stressing over page borders and trying to find the one black cat, no […]

via Providence At Last — Facts in the Case of Alan Moore’s Providence

“Remember the Ifrit”

I sold my second story, and the thrill is just as strong as the first. The Cast of Wonders podcast called for Young Adult science fiction/fantasy stories that evoke a sense of wonder — in 500 words or less. And since this is a podcast, I got to hear my words read by a professional voice actor (and Hugo award-winning author), which is pretty cool.

This is a short one. I hope you enjoy “Remember the Ifrit.” (If you’re in a hurry, my story starts around the 6:00 mark on the audio.)

http://www.castofwonders.org/2017/04/episode-242-little-wonders-11-flash-fiction-contest-finalists/

Writer’s Endnotes

An image that stuck with me from childhood was from an episode of Cosmos, where Carl Sagan hypothesized what kind of life might exist in the atmosphere of a gas giant planet like Jupiter. (I’m pleased to see that the segment clip is up on YouTube.) It was only after I’d written the first draft of “Ifrit” that I realized I was stealing from Carl Sagan. Let’s call it a tribute, instead.

humpback_whale

When the family went on a whale-watching cruise off Whidbey Island in Washington State, I resolved to experience it with my daughter, directly, with no camera involved. But my resolve faltered when the Humpback we’d been following began speeding just under the water’s surface. I had gotten my phone out of my pocket and started taking a video when she breached out of the water and then fell with a tremendous SPLASH. I never got her in frame — I wanted to see it with my naked eye — but in the audio, one can hear the family shouting and whooping, and mine was the loudest voice on the boat. That cruise, and sharing the experience with my daughter, was the first thing I thought of when I began considering topics of a story with a “sense of wonder.”

My cousin-in-law did manage to take a perfect picture of the whale, so I got the direct experience and a memento. I think it’s interesting that it’s not the photo that best recalls that sense of wonder for me, but the audio. The picture represents what I saw; the audio captures how I felt. For the story, I felt that both the experience and its recollection were important, along with the curious modern impulse to interpose devices between ourselves and a wonder in hopes of being able to re-experience it at will. And, of course, the power of an experience shared was the most important of all.

 

 

After Ozymandias

ozymandias

Two thousand years hence, a traveler

In this now-antique land

Spies a statue, vast, translucent, and suffused

With sunlight, standing legs astride and arms

Akimbo in the desert sand.

And on the pedestal these words appear:

My name is Erin Riddick, Chemical Engineer;

Look on my Works, ye Seekers, and rejoice

For I found Immortality in undying Plastics!

And just under the sand, crunching and yielding

Beneath the traveler’s feet, lie the artifacts of

An ancient world, inventions and conveniences,

The fruits of man’s ingenuity outlasting

Even those of God’s.

Thin Mints

Weight Watchers Girl Scout Cookies

He slapped the last box of Thin Mints on the table while we stared. “I deserve these cookies,” he said. “I need them.”

The rest of our Weight Watchers group looked down at our shoes, unable to meet his eyes. The second hand on the clock was deafening.

He grunted. “I’ll come back for these when I want them, not need them.” He turned and stalked out the door. The exhalation of held breaths drowned out the ticking clock.

His footsteps echoed down the hall. I eased a sleeve of Thin Mints from the box and stuffed it into my pocket.


 

I took a pair of Thin Mints from their sleeve — one serving — and threw the box into the trunk of my car. Then I drove. I started with a nibble, that became a bite, and before I could swallow, an entire cookie was in my mouth. The remaining one hung like a cigarette between two fingers, my hands on the wheel.

I only had to make it to the expressway. Then there would be no pulling over. One serving.

I took the barest nibble and held it in my mouth until it dissolved. The morsel of minty chocolate sludge stuck to the roof of my mouth while my esophagus yearned. The pleasure came not just from the taste on the tongue, but in the total act of consuming. To swallow was a release, but the hunger returned like a tide, moments after its ebb.

The on-ramp. Crumbs fell from the bitten cookie and with them came tiny pangs of loss.

The remaining Thin Mint went into my mouth the moment I merged into traffic. Chew, savor, swallow. The pleasure was too small, too fleeting, too sharp. I stomped the accelerator and looked for gaps in the next lane. Flipped the radio presets as a distraction. The Stones came on the Oldies station. I turned it up and sang along at the top of my lungs.

‘Cause I try, and I try, and I try, and I try

I can’t get no

I can’t get no

Satisfaction

The rhythm of the traffic and the music carried me. My mind drifted. The taste in my mouth faded, became a ghost. A memory. I thought about the box in the trunk, and I felt… nothing.

I had won.

I took the long way home.

 

 

How To Be a Tweet Journalist

As public figures in entertainment and government take to Twitter, the public gains a unique form of access to that person’s thoughts, raw and in real time. While you may think that such directness of communication relegates the internet journalist to the role of merely interpreting and contextualizing such messages like some chin-stroking intellectual, you would be wrong. There remains a booming industry of Twitter journalism that simply republishes the public figure’s words — and monetizes it. You too can get started as a Tweet journalist by following an easy formula.

Banner Ad

Begin with a banner ad, to make you some money.

Headline

Also known as the “clickbait,” your headline should allude to the content of the public figure’s Tweet without revealing it. Consider a headline of the format:

{Public figure} Absolutely Destroys {subject of Tweet} In a Single Tweet

There is no need to qualify that the subject of the Tweet wasn’t actually destroyed. It’s implicit in this journalistic form.

Ad Space

This is a good place to put another ad. Now you have two above-the-fold ads to make you some money.

Links To Related Content

This is the place to put some links to other “articles” on your site that have some of the same words. Or fall into some of the same categories. Or relate in more creative ways — have fun with it. Your SEO manager will mutter something about bounce rate, and Google liking richly interlinked content, but all you have to know is, the more Google likes you, the more money you can make on your ad space, and that’s why you’re engaging in this pointlessness, after all.

Background

Okay, you can’t get away with having no interpretation or contextualization. This is where you write one or two paragraphs of background for the Tweet, informing your readers if this is a singular event or part of an ongoing feud. Here, you may give more detail than your headline about what the public figure’s 140 characters will say, but don’t be too clever about it. The Tweet is the headliner here, and your exposition is just the opening act. Check your ego and bons mots at the door.

Ad Space

What, you thought you were done? If your reader has scrolled down this far, then you’ve done your job. And you should be paid for it.

The Tweet

Embed that Tweet. And hope its author doesn’t delete it.

The Future

An ambitious Twitter journalist will want to explore the ramifications of the Tweet going forward. And in the world of Twitter, “going forward” should never mean more than 24 hours. Here is the place to embed the zestiest of @-replies, quoted retweets, and subtweets. Give your readers a 1-stop shop (besides Twitter itself) of the conversation around the public figure’s Tweet. But don’t spend more than 15 minutes of research. Twitter Journalism is a fast-paced business.

Join the Conversation

Invite your users to post to your message board and discuss the topic. 99% of the posts will be garbage or worse, but that doesn’t matter. If you’re using a message board solution that is crawlable by the Google bots, then you’re getting the SEO juice of dynamically changing content, and maybe even some worthwhile “user-generated content” (where readers write things on your site that others may want to read, and you get paid for it).

Ads, Ads, and More Ads

Go for broke. Add a block of those links to “related articles” that another website will pay for you to host. Put in some more animated ads, maybe a pop-up, auto-playing video to punish readers who scroll this far down. The show is over. Roll credits.

 

 

The Trump Plaza Massacre

Chicago witches, along with witches across the nation, are gathering to cast a “binding spell” on the Trump Administration. This is true. But what happened after the first ceremony is a darker tale, one that is officially being called FAKE NEWS.

witches-against-trump

The President realized immediately that he was being constrained. His Executive Order-signing pens kept running out of ink, and he found himself unable to utter complete sentences — which in itself was not unusual, but now, his staff could not decipher his commands. “Steve,” he called out. “I need you.” The shadow the President cast on the Oval Office wall detached itself, slithered along the floor, and wrapped itself like a snake around the Leader of the Free World. Its head stretched over the President’s left shoulder, and pressed a phantom ear to his lips. “I’m being attacked,” the President whispered. The shadow flowed away and snaked its way into the greater shadows at the far side of the room, where it vanished.

In his sanctum at the top of Trump Tower, Steve Bannon chuckled without mirth as a tendril of shadow entered his nostril. For a moment, his eyes clouded with darkness, and he nodded. “Witches and hedge magicians, thinking they know power…” He went to the window, and opened it with a gesture. Outside, the sun shone in a cloudless sky. He frowned. Something passed in front of the sun, rippling as it ate the light. Rising from the far horizon, black vapors streaked up into the sky. As the darkness grew, so did Bannon’s power. The shadows cocooned him, and he transformed. A huge, shapeless, pale thing rose into the darkening sky like a misshapen balloon, and then sped westward.

The Chicago witches swayed in their circle in Trump Plaza. There was a stench on the wind from the east, and waves of nausea hit them. The witch Kloë looked up at the darkening sky, and said to her Sister, “Is… is that us?”

“No,” Breeannaugh said. “There’s someone… else…” She cast her mind skyward, and braving the ethereal miasma that tainted the air, she quested for the source of the disturbance. Her mind met something formless, terrifying, and radiating hideous power. Her body burst like rotten fruit dropped from a height. The witches’ hex snapped and recoiled, hitting the coven with a backlash of magic. With a single voice, they screamed.

A mass of pale flesh slopped onto the ground in the witches’ broken circle. It rose on tendon and sinew from its amorphous roiling. Steve Bannon’s face emerged from a sheath of quivering skin. It grinned. Qaitlynn vomited noisily.

“Hello Snowflakes,” said the thing with Bannon’s face. I still have plans for the President. You will not meddle in matters so far beyond you.”

Elyyzah’s fingers closed on her knife hilt and she whispered a curse, the strongest she knew. Then she leaped at the Alt-Right horror. Her blade sunk to the hilt in Steve Bannon’s gelatin-soft forehead. He smiled. “Nevertheless you persist?” he whispered. She herd mockery in the thing’s voice. And… admiration? She recoiled from the thought of being admired by this thing.

“FLY YOU FOOLS!” Elyyzah screamed to her coven as Bannon slithered over her and began to feed.

Far to the east, from his golden toilet, the President tweeted that the Left, still unable to get over his landslide victory, had resorted to witchcraft to defy the will of the people. Sad!

Hours later, a furious Sean Spicer condemned the dishonest media for its lies about the Administration’s dealings with dark forces, and about the very existence of the Massacre at Trump Plaza.

 

Poisoned Arrows

Cupid lay dying

His bow snapped in twain

At my feet he lay writhing

And moaning in pain

An eye for an eye

So Cupid did pay

For sending his poisonous arrows

My way

My blood is afire. The wound is just a nick, but poison throbs hot in my shoulder. My Queen is in anguish. The poison is taking her too. Stand. Nock. Draw. Hold. Loose. I know failure before the arrow leaves my bow. The demon flaps its wings and sends another shaft into my thigh. Cherub it may seem, but it is the deadliest archer I’ve faced. Again I draw. “For my King,” I whisper. My arrow finds sinew through feathers, and the demon tilts and spirals low. I leap, grasp its foot, and drag it to earth.

I snap its wings like dry branches, and kick its bow from its reach. Its youthful, curl-framed visage belies hideous strength, and we grapple as I strain for my sword.

“Lancelot!”

I glance to Guinevere — my Queen — and too late i see the arrow in the demon’s fist. It pierces my heart. My very soul catches fire.

I am vanquished. My foe is gone. Guinevere cradles my head in her lap. Tears stream down her cheeks, falling like sparks on my fevered brow. The poison roars in our blood. I can feel it in her, throbbing in time with mine. She shakes her head, denying something unseen.

My mouth is sere. Her lips are pink, parted, and — suddenly I learn — impossibly soft. I mustn’t. But we are twin bonfires consuming each other, uncontrolled. I try to fix in my mind the image of my King, but the thought blackens, curls, and disappears in bitter smoke.