How Platform-User Power Relations Shape Algorithmic Accountability: A Case Study of Instant Loan Platforms and Financially Stressed Users in India

By Divya Ramesh, Vaishnav Kameswaran, Ding Wang, Nithya Sambasivan. 2022

57 citations recorded by the source corpus.

Accountability, a requisite for responsible AI, can be facilitated through transparency mechanisms such as audits and explainability. However, prior work suggests that the success of these mechanisms may be limited to Global North contexts; understanding the limitations of current interventions in varied socio-political conditions is crucial to help policymakers facilitate wider accountability. To do so, we examined the mediation of accountability in the existing interactions between vulnerable users and a ‘high-risk’ AI system in a Global South setting. We report on a qualitative study with 29 financially-stressed users of instant loan platforms in India. We found that users experienced intense feelings of indebtedness for the ‘boon’ of instant loans, and perceived huge obligations towards loan platforms. Users fulfilled obligations by accepting harsh terms and conditions, over-sharing sensitive data, and paying high fees to unknown and unverified lenders. Users demonstrated a dependence on loan platforms by persisting with such behaviors despite risks of harms such as abuse, recurring debts, discrimination, privacy harms, and self-harm to them. Instead of being enraged with loan platforms, users assumed responsibility for their negative experiences, thus releasing the high-powered loan platforms from accountability obligations. We argue that accountability is shaped by platform-user power relations, and urge caution to policymakers in adopting a purely technical approach to fostering algorithmic accountability. Instead, we call for situated interventions that enhance agency of users, enable meaningful transparency, reconfigure designer-user relations, and prompt a critical reflection in practitioners towards wider accountability. We conclude with implications for responsibly deploying AI in FinTech applications in India and beyond.

Subfields: Information Systems, Management Information Systems, Safety Research

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Information Systems

In Management Information Systems

Andrew Smart

In Safety Research

Citations in the corpus

PaLM 2 Technical Report (2023, background): “, 2021), and additionally evaluate memorization as a proxy for potential privacy harms (Ramesh et al., 2022).”
Sociotechnical Harms of Algorithmic Systems: Scoping a Taxonomy for Harm Reduction (2022, background): “Here, privacy violations may reflect more traditional conceptualizations of privacy attacks or security violations [79, 105] and privacy elements beyond what may be protected by regulations or under the traditional purview of a privacy officer [138, 180].”
Sociotechnical Harms: Scoping a Taxonomy for Harm Reduction (2022, background): “Privacy violation occurs when technology design leads to diminished privacy, such as enabling the undesirable flow of private information [159], instilling the feeling of being watched or surveilled [160], and the collection of data without explicit and informed consent [100].”