refute_causal_structure

[bernddude]

Hello , i discovered recently your new sub-package gcm, i love it its a great idea. However i am struggling a little bit to understand the details of some of the modules. For example in the refute_causal_structure module i find that my DAG and data set end up being refuted. I tried to inspect the results and noticed a few of the local_markov_test results ended with success ~ False. Now my question is where do i go from here. I do know for a fact from my knowledge about the data that my DAG is sensible for the data. So does the result imply that i should NOT trust the results of DoWhy's gcm package ? If so how do i correct the DAG ? It has far to many components to perform that task via trial and error ?

[bloebp]

Hi,

the current version of the refute_causal_structure method is a bit too 'strict' here. As for now, I would not over-interpret the rejection, just double check the incoming edges to the nodes where you get False. Generally, the local_markov_test gives some indication whether the assumed graph structure captures the conditional independencies in the data correctly. Practically, however, there can still be issues with the used statistical test or a too small sample size etc., which could lead to a false rejection of the graph. The good news is: We are currently working on completely replacing the method with a new one that provides much more detailed insights and not only a binary "reject or not" decision. We hope to bring this in in the next 1-2 weeks.

[bernddude]

Thank you so much for your detailed response Patrick, that makes sense. To understand this a bit better, does a False in the local_markov_test suggest that despite the presence of the arrow the data lack a conditional dependency ? Is that's what you are checking for ? I do understand your comment about lack of data, in my case i didn't think it's the reason since the data set is about 250k sets and the all variables are booleans.
I am excited to check out your update, if you like any feedback please let me know , thx Bernd

[bernddude]

Thx a lot Patrick, i understand the example. You make a good point here, the data are tricky in the sense that there are lots of direct and indirect interactions. It's actually part of why in my view the approach you take is so valuable. I can 'crow bar' everything using some NN however that would loose most o the insight into what is happening.
I will definitely look into your suggestion.
It does actually reminds me to a question i am working on right now. I tried to us ether counterfactual test. However i am having a bit difficulties with the Boolean data set. So far i only got it to work using something like this
Icausal_model = gcm.InvertibleStructuralCausalModel(DAG)

auto assign mechanisms

#gcm.auto.assign_causal_mechanisms(Icausal_model, data)
root = ['MRR','BEL','Squeeze','Hot','Cold']
for n in nodes:
if n in root:
Icausal_model.set_causal_mechanism(n, gcm.EmpiricalDistribution())
else:
Icausal_model.set_causal_mechanism(n, gcm.AdditiveNoiseModel(gcm.ml.create_random_forest_regressor()))
#Icausal_model.set_causal_mechanism(n, gcm.AdditiveNoiseModel(gcm.ml.create_linear_regressor()))
#Icausal_model.set_causal_mechanism(n, gcm.PostNonlinearModel(gcm.ml.create_linear_regressor()))
#Icausal_model.set_causal_mechanism(n, gcm.AdditiveNoiseModel(gcm.ml.create_logistic_regression_classifier()))

2. Fitting the PCM to the data:

gcm.fit(Icausal_model, data)

However this results in continuous output variables . Which i am running through a logistic switch function to get the final state
p/s i realize this is a separate question. If you prefer i can make a separate post to avoid clutter ... thx Bernd

[bloebp]

I would probably regard these as 'categorical' data and convert the values to strings explicitly (e.g., data['MRR'] = data['MRR'].astype(str) for a column or just data.astype(str) for the whole data set). Then, you can assign a ClassifierFCM:
Icausal_model.set_causal_mechanism(n, gcm.ClassifierFCM(gcm.ml.create_logistic_regression_classifier())) or gcm.auto.assign_causal_mechanisms(Icausal_model, data) for an automatic assignment. These would give you categorical data back. Otherwise, you are currently treating it as a regression problem.

One issue is now, however, counterfactuals require to re-cover the noise, which is currently not possible with categorical non-root nodes. However, you can consider interventions instead, they also work with categorical data. Depending on your use case, interventions might also give some good insights.

PS: An InvertibleStructuralCausalModel is strictly speaking not compatible with ClassifierFCMs, should be a StructuralCausalModel instead. But it also won't throw an error at the moment

[bernddude]

Thank you again Patrick, that is pretty much what i did (use intervention).
My question was exactly b/c of the noise model used for the counterfactual calculation.
It does seem that the 'trick' i used with the random forrest regressor and to post process the results as probability of ending up in the positive or negative state provides sensible results though. It was interesting that the logistic function with a hard switch didn't work at all . Basically it concluded no influence from anything since the individual values hardly ever moved enough. However once i assigned it a probability of state change the results look quite consistent with known test cases .
Thx again, i really appreciate your help

[bloebp]

Great! Let me know if you run into any more issues :) That is great feedback for improvement.

[bernddude]

I totally value the help Patrick. If this kind of feedback helps, i am very happy to provide it

[eeulig]

Hi @bernddude! In #930 we added new methods to evaluate some user-given graph on data (the ones @bloebp mentioned in his comment). In particular, we added a function falsify_graph to compare the result of testing local Markov conditions on the given graph with the results on random node-permutations of the given graph. By this we can measure whether the number of violations (rejections) is significantly lower than for a random node-permutation. We also added functions to visualize for which nodes the violations occured and to suggest the removal of certain edges. In this example notebook you can find a few examples that are hopefully helpful to understand how the method works and how to apply it to your data.
We would be happy about any feedback or suggestions to improve these methods further. If you have any questions, please let me know!