Disclaimer: I was a little drunk on power (calculations) when I wrote this, but it’s me figuring out that econometrics is something I might want to specialize in!
I think I just figured out what I want to do with the rest of my career.
I want to contribute to how people actually practice data analysis in the development sector from the technical side.
I want to write about study design and the technical issues that go into running a really good evaluation, and I want to produce open source resources to help people understand and implement the best technical practices.
This is always something that makes me really excited. I don’t think I have a natural/intuitive understanding of some of the technical work, but I really enjoy figuring it out.
And I love writing about/explaining technical topics when I feel like I really “get” a concept.
This is the part of my current job that I’m most in love with. Right now, for example, I’m working on a technical resource to help IDinsight do power calculations better. And I can’t wait to go to work tomorrow and get back into it.
I’ve also been into meta-analysis papers that bring multiple studies together. In general, the meta-practices, including ethical considerations, of development economics are what I want to spend my time working on.
I’ve had this thought before, but I haven’t really had a concept of making that my actual career until now. But I guess I’ve gotten enough context now that it seems plausible.
I definitely geek out the most about these technical questions, and I really admire people who are putting out resources so that other people can geek out and actually run better studies.
I can explore the topics I’m interested in, talk to people who are doing cool work, create practical tools, and link these things that excite me intellectually to having a positive impact in people’s lives.
My mind is already racing with cool things to do in this field. Ultimately, a website that is essentially an encyclopedia of development economics best practices would be so cool. A way to link all open source tools and datasets and papers, etc.
But top of my list for now is doing a good job with and enjoy this power calculations project at work. If it’s as much fun as it was today, I will be in job heaven.
Ӧzler summarizes his main points quite succinctly himself:
Think about the meaningful effect size in your context and given program costs and aims.
Power your study for large effects, which are less likely to disappear in the longer run.
Try to use all the tricks in the book to improve power and squeeze more out of every dollar you’re spending.”
He gives a nice, clear example to demonstrate: a 0.3 SD detectable effect size sounds impressive, but for some datasets, this would really only mean a 5% improvement which might not be meaningful in context:
“If, in the absence of the program, you would have made $1,000 per month, now you’re making $1,050. Is that a large increase? I guess, we could debate this, but I don’t think so: many safety net cash transfer programs in developing countries are much more generous than that. So, we could have just given that money away in a palliative program – but I’d want much more from my productive inclusion program with all its bells and whistles.”
Usually (in an academic setting), your goal is to have the power to detect a really small effect size so you can get a significant result. But Ӧzler makes the opposite point: that it can be advantageous to only power yourself to detect what is a meaningful effect size, decreasing both power and cost.
He also advises, like the article I posted about yesterday, that piloting could help improve power calculations via better ICC estimates: “Furthermore, try to get a good estimate of the ICC – perhaps during the pilot phase by using a few clusters rather than just one: it may cost a little more at that time, but could save a lot more during the regular survey phase.”
My only issue with Ӧzler’s post is his chart, which shows the tradeoffs between effect size and the number of clusters. His horizontal axis is labeled “Total number of clusters” – per arm or in total, Bert?!? It’s per arm, not total across all arms. There should be more standardized and intuitive language for describing sample size in power calcs.
Two weeks ago, I had my first opportunity to climb outdoors. My friends were going to Hell’s Gate – the national park two hours outside Nairobi that inspired much of the Lion King film.
I have been afraid of heights my whole life. That fear is one of the main reasons I’ve never gone rock climbing outdoors. In a rock climbing gym, the controlled environment feels like a pretty safe space to dangle from a rope two to three stories up. But when I get near the edge of a cliff, I feel like I suddenly have to fight the urge to leap into the void.
This really freaked me out when I was younger, even though the fear was tinged with a sense of exhilaration, too.
Aside: Recently (read: five minutes ago), I learned this urge is called the “high place phenomenon.” In one study on the feeling, researchers found the desire to jump wasn’t correlated with suicidal thoughts and was instead correlated with “anxiety sensitivity.” Anxiety sensitivity is essentially how anxious being anxious makes you – specifically how reactive you are to the physical sensations of your body telling you it’s in danger (like the quaking in your knees as you look over a cliff).
It was one of the most beautiful days I have ever experienced. It had been rainy all week (we’re just wrapping up the rainy season in Kenya), but the day we were climbing was all sunshine and scudding clouds.
We hired a climbing guide to set up two top ropes so that we could belay ourselves after that. We had one easier climb and one harder one. Later, another climber showed up and set up another climb and we moved the easier rope to another wall. I was able to try all four and got to the top of three. The fourth had an overhang and was the last one I attempted. I got my arms onto the overhang but couldn’t haul myself over the top that late in the day.
After we “cleaned” the routes (removed the equipment we had used for top-roping), we drove through the rest of the park to reach some sulphur hot springs on the opposite side. The whole landscape was wide open and gorgeous.
At one point a giraffe just started running alongside our car. It was magical.
10/10 experience and made me really want to climb more!
Two weekends ago, I went on one of the most beautiful hikes I’ve ever been on – Elephant Hills in the Aberdares.
Unfortunately, the hike was 3 hours longer than advertised, we ran out of water, we had to hike in the dark (with hyenas??), and I got altitude sickness and threw up. Otherwise, best hike ever.
“The Aberdares? In the rainy season? Are you crazy?!” – My boss when she heard about our trip at work the next day
Mikala found a group hiking Elephant Hills – a hike we’d all heard was crazy beautiful – and invited Brooke and me. I added Nick and Alice from work to the hiking crew. Women Who Hike Kenya organized the buses, park fees, park rangers, and “personal photographer” (which is why I have so many pics from the hike – I didn’t get my phone out to take any pics until we reached the peak).
The best section going up and down was the bamboo forest we passed through. On the way up, we thought this was the middle third of the hike up – it was more like one of the middle tenths of the climb.
We started out by criss-crossing a muddy road, then followed the road through potato farms, through a pine forest, through an electric fence to keep the game out, through more trees, and along another open grassy section before reaching the bamboo part. On the way down, these pre-bamboo sections all merged into one quick burst in our minds… instead, it kept going and going and going. The bamboo section was just a vertical shoot up a mud slide between gorgeous bamboo shoots – lots more falling.
After the long bamboo section, we took a quick break on a grassy knoll. A bunch of us thought it was the elephant’s head, so we ate all our lunch. But then on we went.
The next section was just a muddy stream of squelching mud sprinkled with safari ants (vicious biters, but I thankfully wasn’t bit). We also thought this was the last section…
…until we arrived at another grassy section leading us up into the clouds.
After that, I started feeling really sick and decided I had to just power through to the peak. The clouds faked us out at least 5 or 6 times before I finally made it. As soon as I reached the top, the clouds parted and we were treated to incredible 360 views. I was too busy dying of altitude sickness to notice at first. I recovered a bit and ate some of my leftover “I’m not going to Liberia for two months!”chocolate cake from Java.
I was desperate to get down from altitude (about 12,500 ft at the peak, up nearly 3000 ft from the trailhead), so I tried to keep up with Alice and Nick. They hadn’t been feeling the altitude sickness like Brooke, Mikala, and me. Pretty quickly, though, I had to stop and almost started crying my head hurt so much. I let the rest of the summiters pass me by until Brooke and Mikala reached me and rallied my spirits.
Still, Mikala and I were hit pretty hard and it was slow going. Meanwhile, Brooke was calculating how long it would take to get down, comparing that timeline to when it would get dark, and debating whether it was better to push us to go faster when we were feeling so crappy or to be in a national park with wild animals after dark. Actually, at that point, we were more worried about being in that dense, slippery bamboo in the dark.
My head was pounding, I felt hopeless but knew I had to keep going, and my legs were quaking. I’ve never seen my legs shake as hard as they did each time I paused to take a sip of water or breath more deeply. I’d been sitting on the idea that I needed to throw up for about 15 minutes when I finally sped up a bit, turned off the trail and puked. After that, I felt AWESOME. My legs were still shaking, but now my head wasn’t pounding.
We made it to the first grassy knoll where we’d eaten lunch. Happily, another group of friends on the hike had over-prepared with extra gatorades and lent us some. We refueled and then plunged back into the bamboo forest.
We had some great jungle-woman moments slipping and sliding down the increasingly dimly lit bamboo section. The bamboo on the edge of the path were key; we swung between them rather than trust our weight on the muddy slope.
By the time we made it to the end of the bamboo, we were euphoric and wanted to power through the final spurt. But by the time we hit the pine tree forest again, it was already dark. At that point, one of the more experienced hikers who had done Elephant Hills seven or eight times before started getting antsy. He kept hiking super fast but also yelling at the few stragglers to stay with the group and warning everyone about hyenas.
At that point, though, I was actually in a great mood. I had a stick to wave menacingly at the dark edges of the path, I wasn’t at a crazy altitude anymore, and I knew where we were and how to get back.
In all, we hiked 12 miles, straight up and straight down through deep mud. No switchbacks in the Aberdares, apparently! I was out on the trail from 9 am to 7 pm.
10/10 would do again… but maybe only through the bamboo section. And probably not in the rainy season!
I recently read Brené Brown’s Daring Greatly. The book presents Brown’s research, but it can feel more like a personal guidebook to tackling issues of vulnerability and shame.
Because the research has a conversational feel, it’s hard to understand how much of the book is based in research and how much in Brown’s individual experiences. She weaves in personal stories frequently, often to demonstrate a prickly emotional experience that was common across her interviews. But when I reached the end of the book, I wanted to know how she drew these theories from the data. I’ve only worked sparingly with qualitative data: how does one “code” qualitative data? How do you analyze it without bringing in all sorts of personal biases? How do you determine its replicability, internal and external validity, and generalizability?
Ingeniously, Brown grounds the book in her research methods with a final chapter on grounded theory methodology. Her summary (also found online here) was a good introduction to how using grounded theory works and feels. But I still didn’t “get” it.
So I did some research.
Brown quotes 20th century Spanish poet Antonio Machado at the top of her research methods page:
“Traveler, there is no path. / The path must be forged as you walk.”
This sentiment imbued the rest of the grounded theory (GT) research I did. Which seemed bizarre to a quant-trained hopeful economist. I’m used to pre-analysis plans, testing carefully theorized models, and starting with a narrow question.
Grounded theory is about big questions and a spirit of letting the data talk to you.
Founded by Barney Glaser and Anselm Strauss in 1967, GT is a general research methodology for approaching any kind of research, whether qual- or quant-focused. When using GT, everything is data – your personal experiences, interviews, mainstream media, etc. Anything you consume can count, as long as you take field notes.
Writing field notes is one of the key steps of GT: coding those notes (or the data themselves – I’m still a little blurry on this) line-by-line is another. The “codes” are recurring themes or ideas that you see emerging from the data. It is a very iterative methodology: you collect initial data, take field notes, code the notes/data, compile them into memos summarizing your thoughts, collect more data based on your first learnings, code those, compile more memos, collect more data…
Throughout the whole process, you are theorizing and trying to find emergent themes and ideas and patterns, and you should actively seek new data based on what your theories are. You take a LOT of written notes – and it sounds like in the Glaserian tradition, you’re supposed to do everything by hand. (Or is it just not using any algorithms?)
Brown describes the data she collected and her coding methodology:
“In addition to the 1,280 participant interviews, I analyzed field notes that I had taken on sensitizing literature, conversations with content experts, and field notes from my meetings with graduate students who conducted participant interviews and assisted with the literature analysis. Additionally, I recorded and coded field notes on the experience of taking approximately 400 master and doctoral social-worker students through my graduate course on shame, vulnerability, and empathy, and training an estimated 15,000 mental health and addiction professionals.
I also coded over 3,500 pieces of secondary data. These include clinical case studies and case notes, letters, and journal pages. In total, I coded approximately 11,000 incidents (phrases and sentences from the original field notes) using the constant comparative method (line- by- line analysis). I did all of this coding manually, as software is not recommended in Glaserian-grounded theory.” [emphasis mine]
The ultimate goal is to have main concepts and categories emerge from the data, “grounded” in the data, that explain what main problem your subjects are experiencing and how they are trying to solve that problem. For example, Brown’s work centers on how people seek connection through vulnerability and try to deal with shame in various health and unhealthy ways. She started with this big idea of connection and just started asking people about what that meant, what issues there were around it, etc. until a theory started to arise from those conversations.
You’re not supposed to have preexisting hypotheses, or even do a literature review to frame specific questions, because that will bias how you approach the data. You’re supposed to remain open and let the data “speak to you.” My first instinct on this front is that it’s impossible to be totally unbiased in how you collect data. Invariably, your personal experience and background determine how you read the data. Which makes me question – how can this research be replicable? How can a “finding” be legitimate as research?
My training thus far has focused on quantitative data, so I’m primed to preference research that follows the traditional scientific method. Hypothesize, collect data, analyze, rehypothesize, repeat. This kind of research is judged on:
Replicability: If someone else followed your protocol, would they get the same result?
Internal validity: How consistent, thorough, and rigorous is the research design?
External validity: Does the learning apply in other similar populations?
Generalizability: Do the results from a sample of the population also apply to the population as a whole?
GT, on the other hand, is judged by:
Fit: How closely do concepts fit the incidents (data points)? (aka how “grounded” is the research in the data?)
Relevance: Does the research deal with the real concerns of participants and is it of non-academic interest?
Workability: Does the developed theory explain how the problem is being solved, accounting for variation?
Modifiability: Can the theory be altered as new relevant data are compared to existing data?
I also read (on Wikipedia, admittedly), that Glaser & Strauss see GT as never “right” or “wrong.” A theory only has more or less fit, relevance, workability, or modifiability. And the way Brown describes it, I had the impression that GT should be grounded in one specific researcher’s approach:
“I collected all of the data with the exception of 215 participant interviews that were conducted by graduate social-work students working under my direction. In order to ensure inter-rater reliability, I trained all research assistants and I coded and analyzed all of their field notes.”
I’m still a bit confused by Brown’s description here. I didn’t know what inter-rater reliability was, so I had assumed it meant that the study needed to have internal consistency in who was doing the coding. But when I looked it up online, it appears to be the consistency of different researchers to code the same data in the same way. So I’m not sure how having one person do all of the research enables this kind of reliability. Maybe if your GT research is re-done (replicated) by an independent party?
My initial thoughts are that all GT research sound like they should have two authors that work in parallel but independently, with the same data. Each develops separate theories and then at the end, the study can compare the two parallel work streams to identify what both researchers found in common and where they differed. I still have a lot of questions about how this works, though.
A lot of my questions are functional. How do you actually DO grounded theory?
How does GT coding really work? What does “line-by-line” coding mean? Does it mean you code each sentence or literally each line of written text?
Do these ever get compiled in a database? How do you weight data sources by their expertise and quality (if you’re combining studies and interviews with average Joes, do you actively weight the studies)? -> Can you do essentially quantitative analysis on a dataset based on binary coding of concepts and categories?
How do you “code” quantitative data? If you had a dataset of 2000 household surveys, would you code each variable for each household as part of your data? How does this functionally work?
If you don’t do a literature review ahead of time, couldn’t you end up replicating previous work and not actually end up contributing much to the literature?
And then I also wondered: how is it applicable in my life?
Is GT a respected methodology in economics? (I’d guess not.)
How could GT enhance quant methods in econ?
Has GT been used in economic studies?
What kinds of economic questions can GT help us answer?
Should I learn more about GT or learn to use it in my own research?
This week, one of my favorite podcasts – AdultSh1tfrom Kate Peterman and Kelsey Darraghof Buzzfeed – answered a question from a listener whose sex drive is much higher than that of her long-time girlfriend. The listener said she felt bad and uncomfortable asking for more sex because she doesn’t want to feel “rape-y” or like she’s pressuring her girlfriend into doing sexual things she doesn’t want to do. But she’s not getting the sexual fulfillment she needs right now.
Kate and Kelsey advise even more communication, but also to make sure you’re getting what you need in the relationship. I would add a few things to what they shared:
Start the conversation from a point of, “I love you and I want to figure this out together. We seem to need different things and I want to understand why you need what you need and explain why I need what I need.” It is super scary to start a conversation with a long-time partner that you know could lead to some really painful and possibly break-up-inducing discussions. Phrasing it like you’re tackling an issue together makes it seem less intimidating to get into it at a deeper level.
Be careful to not attack her perspective/experience. I think this is especially challenging because there’s an assumption that everyone wants to have sex inherent in U.S. culture. Being sensitive to the fact that she might be ready to get defensive about that assumption (just like you could be sensitive about the assumption that having a higher sex drive is “sinful”) can help. Avoid approaching it like there’s a problem with her. It’s a problem between the two of you.
Share that you don’t want to feel like you’re pressuring her, that you’re struggling to approach the issue, and that that’s a point of pain for you. Specifically share that it can make you feel unwanted, unsexy, and confused. Maybe this can lead to her sharing ways that you can introduce sexy times without it being pressuring, or what specific language y’all can use to discuss whether to get it on in the future. Also, ask her to tell you how these situations make her feel – is she uncomfortable? Annoyed? Sad? Frustrated? It might also be hard for her to know she’s not fulfilling all your sexual needs. My good friend Annaji introduced me to the powers of the “I” statement. If you keep it about how you feel, that acknowledges there’s room for misinterpretation and offers the other person space to clarify where they’re coming from, too.
Once you’ve both been able to share how y’all are feeling about your sex life, then you can talk about next steps more freely and as a team. Maybe you can brainstorm some solutions or compromises. Maybe, even though it’s scary, you both will find you really do need a partner whose sex drive matches your own more closely. Even though it’s a tough decision to make after so many years together.
I was once in a relationship where I found out almost 2 years in that my partner was unhappy with how much sex we’d been having. He felt like we had sex too often. He told me that our intimacy cut into his time to do personal projects that were important for him to be fulfilled. This was partially because we would sleep over in each others’ dorm rooms a lot and go to bed early, cutting into hours he would have previously used to create.
It was really painful to hear that he felt pressure to have sex, even if he said that pressure wasn’t coming from me so much as from society. I still worry that he did feel pressure from me. Maybe, as I discuss below, he just couldn’t tell me he was uncomfortable.
We did break up shortly after this revelation, which came at a complicated time in our relationship for other reasons. I’m not sure how we would have dealt with our mismatched sex drives given more time. Thankfully, we still have a good relationship as friends. And I think that’s in large part due to the fact that we did have honest conversations about our sex life before the break-up.
Men’s experiences: the same, but different
This episode also deepened my thinking on the Beautiful / Anonymous episode I shared earlier this week, where a man called in to talk about how he was sexually assaulted three times in his life, twice by women. I had a moment at work today to discuss it with my colleague who recommended the episode. One of our big takeaways was that it was amazing how familiar the caller’s description of his doubt, shame, and struggle sounded.
Most women have heard female friends’ stories of sexual assault or experienced it themselves. Through most stories, there are common threads: feeling unable to react fast enough or strongly enough, doubting whether we truly couldn’t have done anything, wondering if the other person genuinely thought it was okay and just missed our signals, whether the signals we sent were strong enough. All of that self-doubt I have heard expressed by numerous women – it was a revelation to hear a man share the same doubts.
Anyone can have their bodily autonomy and safety violated. Even if you know that in your brain, hearing this man’s story makes it stick in your soul in a new way. I think all of us still carry with us ingrained messaging about men’s relationship to intimacy, sex, and violence. Those narratives lead us to make assumptions that can really hurt other people; this podcast invites us (women) to examine our part in perpetuating these assumptions about gender and sex.
It’s hard to think about, because it really complicates narratives that are easier to keep clean cut. Women are victims, men are violent, sex-driven animals. We (feminists) know it’s a gross simplification, but it’s still so tempting when the statistics are that most victims of sexual violence are women, and most perpetrators are men. That narrative can even be comforting/validating on some level for women who are violated in that way – you’re not alone, this happened to us too, we’re here for you, it’s not your fault.
While the male caller shared these same doubts and feelings of shame and guilt as so many women, the way he expressed the doubts was also telling. My colleague and I both marked that he was bewildered by those feelings and without a narrative to put them into that accounted for both his masculinity and his vulnerability.
My female friends and I, on the other hand, can see how our own experiences fit into the larger societal phenomenon of violence against women. Each of our experiences is intensely personal and can feel isolating. Yet when we’ve been able to talk to each other about those experiences, we can wryly see it as part of “womanhood” in our culture. A terrible yet shared burden. We all contribute a piece to the larger narrative. And, we can also share in the new narratives that are rising about self-care, how to survive & thrive after sexual violence, how to find sisterhood in this massive, horrifying phenomenon.
It is both to men’s privilege and to the caller’s disadvantage that he is not part of this narrative. This podcast called me to think about ways in which I perpetuate bad myths about gender, sex and violence, and ways in which I can bring non-female survivors of sexual assault into the sisterhood component of the narrative I hold.
Ran into my colleague Hanna at our corner veggie market. Bought 2 green peppers, 3 tomatoes, and 3 onions to make dal, plus 3 bananas to add to smoothies; all together, that cost 140 KES, which is about $1.40.