The Behavior Modality: Studying Human Behavior In The Wild

Our aim is to explore the Behavior Modality. This involves explaining, predicting, and optimizing human behavior in the real world, especially, from the context of marketing, advertising, and computational social science.

Behavior as a modality¹ occurs in the process of communication. Communication includes all of the procedures by which one mind may affect another [1]. This includes all forms of expression, such as words, gestures, speech, pictures, and musical sounds.

Communication is composed of five modalities (shown in the diagram below (Fig. 1)): (1) Speaker, (2) Content, (3) Channel, (4) Receiver, (5) Behavior (Effect). One can also include two other (minor) modalities to these five: (6) Time of sending (7) Time of receiving. These modalities may vary independently of each other [2], [3], [4] and carry signals about each other [5], [6] [7]. The message as a modality carries information from the communicator to receiver and encodes information generated by the communicator. Similarly, behavior (aka effect) as a modality carries information from the receiver and encodes information generated by the receiver. This is often a continuous cycle, where behavior generated in the previous cycle becomes the message of the next cycle, thus forming a conversation.

Therefore, from the above seven modalities, we get the following problems:

• Behavior Simulation: How will a certain audience behave for a certain message sent by a certain sender at a sender time over a certain channel?
• Customer Behavior Optimization with Content as a Hook: How should a message be constructed such that it gets a certain behavior from a certain audience?
• Audience Optimization: To whom should a message be sent in order to get a certain effect?
• Sender Optimization: Who should send a message to get a certain effect?
• Time Optimization: What time and with what frequency should a message be sent so that it gets a certain effect?
• Ultimately, how much free will is there with humans when making decisions?
• ....

A motivation of why we are work on the behavior modality is captured in a small blog by Yaman.

Our Contributions:

• LCBM: Large Content and Behavior Models to Understand, Simulate, and Optimize Content and Behavior
• BOIG: Behavior Optimized Image Generation
• Long-Term Ad Memorability: Understanding and Generating Memorable Ads
• Persuasion Strategies in Advertisements
• Synthesizing Human Gaze Feedback for Improved NLP Performance
• A Video Is Worth 4096 Tokens: Verbalize Videos To Understand Them In Zero Shot
• LLaVA Finds Free Lunch: Teaching Human Behavior Improves Content Understanding Abilities Of LLMs

1. A modality is defined in terms of information, such that a modality is a medium through which information is conveyed [7], [8], [9]. Similarly, a multimodal distribution is defined as having more than one peak in the probability distribution describing the nature of information.