c#database entity-framework twitter linq-to-twitter

Entity Framework throwing an error when adding data without IDs

I've a function which returns a list of tweets. Each element in that list has a lot of data, the tweet's text, location, user information, if it was a retweet, its time, retweet information, picture links, and a lot more data, lists of lists of data in each other.

I am using Linq2Twitter package which provides me with an object called Status which is what I am referring to as a Tweet.

This code fetches the data from my function: var container = await DoPagedSearchAsync(context, this);

My DBSet looks like this:

       public class TweetContext : DbContext
    {
        public DbSet<Status> Tweets { get; set; }
    }

I am trying to add the data with this code:

        using (var vontex = new TweetContext())
        {
            vontex.Tweets.AddRange(container);
            vontex.SaveChanges();
        }

And the following is the list of errors I get at run-time on the line: vontex.Tweets.AddRange(container); :

One or more validation errors were detected during model generation:

DataAcquirer.Models.Status: : EntityType 'Status' has no key defined. Define the key for this EntityType. DataAcquirer.Models.Coordinate: : EntityType 'Coordinate' has no key defined. Define the key for this EntityType. DataAcquirer.Models.Entities: : EntityType 'Entities' has no key defined. >Define the key for this EntityType. DataAcquirer.Models.HashTagEntity: : EntityType 'HashTagEntity' has no key defined. Define the key for this EntityType. DataAcquirer.Models.PhotoSize: : EntityType 'PhotoSize' has no key defined. Define the key for this EntityType. DataAcquirer.Models.VideoInfo: : EntityType 'VideoInfo' has no key defined. Define the key for this EntityType. DataAcquirer.Models.Variant: : EntityType 'Variant' has no key defined. Define the key for this EntityType. DataAcquirer.Models.SymbolEntity: : EntityType 'SymbolEntity' has no key defined. Define the key for this EntityType. DataAcquirer.Models.UrlEntity: : EntityType 'UrlEntity' has no key defined. Define the key for this EntityType. DataAcquirer.Models.UserMentionEntity: : EntityType 'UserMentionEntity' has no key defined. Define the key for this EntityType. DataAcquirer.Models.Geometry: : EntityType 'Geometry' has no key defined. Define the key for this EntityType. DataAcquirer.Models.User: : EntityType 'User' has no key defined. Define the key for this EntityType. DataAcquirer.Models.BannerSize: : EntityType 'BannerSize' has no key defined. Define the key for this EntityType. DataAcquirer.Models.Category: : EntityType 'Category' has no key defined. Define the key for this EntityType. Tweets: EntityType: EntitySet 'Tweets' is based on type 'Status' that has no keys defined. Coordinates: EntityType: EntitySet 'Coordinates' is based on type 'Coordinate' that has no keys defined. Entities: EntityType: EntitySet 'Entities' is based on type 'Entities' that has no keys defined. HashTagEntities: EntityType: EntitySet 'HashTagEntities' is based on type 'HashTagEntity' that has no keys defined. PhotoSizes: EntityType: EntitySet 'PhotoSizes' is based on type 'PhotoSize' that has no keys defined. VideoInfoes: EntityType: EntitySet 'VideoInfoes' is based on type 'VideoInfo' that has no keys defined. Variants: EntityType: EntitySet 'Variants' is based on type 'Variant' that has no keys defined. SymbolEntities: EntityType: EntitySet 'SymbolEntities' is based on type 'SymbolEntity' that has no keys defined. UrlEntities: EntityType: EntitySet 'UrlEntities' is based on type 'UrlEntity' that has no keys defined. UserMentionEntities: EntityType: EntitySet 'UserMentionEntities' is based on type 'UserMentionEntity' that has no keys defined. Geometries: EntityType: EntitySet 'Geometries' is based on type 'Geometry' that has no keys defined. Users: EntityType: EntitySet 'Users' is based on type 'User' that has no keys defined. BannerSizes: EntityType: EntitySet 'BannerSizes' is based on type 'BannerSize' that has no keys defined. Categories: EntityType: EntitySet 'Categories' is based on type 'Category' that has no keys defined.

The class looks something like the following:

using System;
using System.Collections.Generic;
using System.Xml.Serialization;
using LitJson;

namespace LinqToTwitter
{
 [XmlType(Namespace = "LinqToTwitter")]
 public class Status
 {
    public Status();
    public Status(JsonData status);

    public Annotation Annotation { get; set; }
    public List<Contributor> Contributors { get; set; }
    public Coordinate Coordinates { get; set; }
    public int Count { get; set; }
    public DateTime CreatedAt { get; set; }
    public ulong CurrentUserRetweet { get; set; }
    public long Cursor { get; set; }
    public Cursors CursorMovement { get; set; }
    public EmbeddedStatus EmbeddedStatus { get; set; }
    public Entities Entities { get; set; }
    public bool ExcludeReplies { get; set; }
    public Entities ExtendedEntities { get; set; }
    public int? FavoriteCount { get; set; }
    public bool Favorited { get; set; }
    public FilterLevel FilterLevel { get; set; }
    public ulong ID { get; set; }
    public bool IncludeContributorDetails { get; set; }
    public bool IncludeEntities { get; set; }
    public bool IncludeMyRetweet { get; set; }
    public bool IncludeRetweets { get; set; }
    public bool IncludeUserEntities { get; set; }
    public string InReplyToScreenName { get; set; }
    public ulong InReplyToStatusID { get; set; }
    public ulong InReplyToUserID { get; set; }
    public string Lang { get; set; }
    public bool Map { get; set; }
    public ulong MaxID { get; set; }
    public StatusMetaData MetaData { get; set; }
    public EmbeddedStatusAlignment OEmbedAlign { get; set; }
    public bool OEmbedHideMedia { get; set; }
    public bool OEmbedHideThread { get; set; }
    public string OEmbedLanguage { get; set; }
    public int OEmbedMaxWidth { get; set; }
    public bool OEmbedOmitScript { get; set; }
    public string OEmbedRelated { get; set; }
    public string OEmbedUrl { get; set; }
    public Place Place { get; set; }
    public bool PossiblySensitive { get; set; }
    public Status QuotedStatus { get; set; }
    public ulong QuotedStatusID { get; set; }
    public int RetweetCount { get; set; }
    public bool Retweeted { get; set; }
    public Status RetweetedStatus { get; set; }
    [XmlIgnore]
    public Dictionary<string, string> Scopes { get; set; }
    public string ScreenName { get; set; }
    public ulong SinceID { get; set; }
    public string Source { get; set; }
    public ulong StatusID { get; set; }
    public string Text { get; set; }
    public bool TrimUser { get; set; }
    public bool Truncated { get; set; }
    public string TweetIDs { get; set; }
    [XmlIgnore]
    public StatusType Type { get; set; }
    public User User { get; set; }
    public ulong UserID { get; set; }
    public List<ulong> Users { get; set; }
    public bool WithheldCopyright { get; set; }
    public List<string> WithheldInCountries { get; set; }
    public string WithheldScope { get; set; }
 }
}

The above model can not be edited. It comes with Linq2Twitter package.

Solution

This approach won't work. The Twitter API returns objects that are hierarchical and not relational. e.g. Here's a snippet of what LINQ to Twitter consumes:

{
  "retweeted":false,
  "in_reply_to_screen_name":null,
  "possibly_sensitive":false,
  "retweeted_status":{
     "retweeted":false,
     ...,
     "user":{
        "id":41754227,
        ...
     },
     ...
  },
  "contributors":null,
  "coordinates":{
      "type":"Point",
      "coordinates":[
          -122.40060,
          37.78215
      ]
  },
  "place":null,
  "user":{
     "id":15411837,
     ...
  },
  "retweet_count":393,
  ...
}

LINQ to Twitter translates that into a Tweet type and the Tweet type has a few idioms to make it easier to build LINQ where clauses. The Tweet is still hierarchical.

To understand the hierarchical structure of the Tweet, notice that the JSON above has more complex objects under the Tweet, such as retweeted_status (RetweetedStatus in LINQ to Twitter), which is a Tweet type property of the Tweet, and user which is a User type property of the Tweet. Within RetweetedStatus and User are more complex objects.

Toward the problem you're seeing, notice the coordinates property. It does not have an ID as it's primarily a value. Perhaps it would have been better modeled as a struct for it's semantics, and maybe that's one of my take-aways here, but the fact is that it doesn't, and will never, have an ID. If you go through the list of items in your error message, you'll see that most of the errors are for value type objects like coordinates.

One approach is to not persist in a relational model and look for a NoSQL option that will meet your needs. e.g. MongoDB or Microsoft's new CosmosDB - there are countless more choices. I've taken this approach in my own code because, while the Twitter API is much more disciplined these days, it used to change unpredictably. That said, a relational approach might be doable these days. In this approach you can read the raw contents from Twitter like this:

string jsonResults = vontex.RawResults;

Then you can use Json.NET to pull out individual tweets.

Here's what I would do with a relational approach:

Use Automapper. The small amount of time it will take to learn this tool will pay many dividends in not only this project but in the future.
Create your own Tweet object suitable for relational persistence.
Use relational references to refer retweeted statuses back to the original tweet.
Create a User table and create a reference from the Tweet to the User.
For all the value type objects, you have two options: a. If it's a single value, like coordinates, flatten it into the Tweet. b. If it's a multi-value, like entities, create a separate table where the entity has it's own pseudo-key and refers back to the Tweet.

I know you would like to see code. However, this is a huge job and writing code would be comparable to me writing your entire data access layer for you, which wouldn't be reasonable for a forum answer. I would still go the NoSQL route unless there's a hard requirement to go relational. Hope this helps.